Http Hobo

Another blog with web-development tips’n’tricks. | About me

Ctrl + ↑ Later

Ensuring executed script’s parent directories permissions in Unix-like systems

I’ve recently discovered that it’s not sufficient enough to grant +x permission to a script in order to make it executable. Besides that you have to check if script’s parent directories also have execution rights. But it depends on how you call the script.

If you use absolute path, than all the parent directories up to the one in root folder will be checked for having execution rights.

; If you execute the following script, directories "path", "to" and "my" should have execution rights
/path/to/my/script.sh

If you use relative path, than you should grant execution rights to all the directories mentioned in the script’s path. Actually this rule applied to absolute path too.

; Here execution rights are checked for "to" and "my" directories only
cd /path
to/my/script.sh

; This scenario is the same as absolute path - .. refers to "path" folder, so it need to have execution rights here as well
cd /path
../to/my/script.sh

So, summarizing it I would put the rule here again.

If a script is being executed in Unix-like system, all the parent directories mentioned in the script path have to have execution rights.

2017   unix

Using non-npm 3rd-party library with Ionic application

I’m currently working on an Ionic application, which involves Google Maps. I tried native plugin, but had not much luck with it, furthermore it doesn’t work with GeoJson, which I need. Thus I decided to go with Google Maps JS API for now and had to find out the way to use 3rd-party library within Ionic app.

According to Ionic documentation it should be smooth and easy. But this works only for npm libraries. What if you need to use a library which is injected into a project in an old-fashioned way using <script src="..."></script> tag?

The main complication is that when we use pure JS, we add a script to the page and then use the following code to create a map:

var map = new google.maps.Map(document.getElementById('map'), {
  // some options here
});

It wouldn’t work with TypeScript, because TS would be unaware of “google” class, therefore will throw an error during compilation.

That’s what “typings” are needed for. Typings is a set of libraries containing definition of types needed for external libraries, so they can be used with TypeScript.

When I googled the matter, I’ve found the following solution:

npm install typings --global
typings install dt~google.maps --global --save

Don’t use it, correct way today is to use npm module @types/*. In my case it would be:

npm install @types/googlemaps

This is actually described in a documentation for which I’ve provided a link above.

But that’s not a big deal. Big deal is about how to let your project know about these typings. Remember – our library is not a NPM module. By default it will say Cannot find namespace “google”. I’ve googled and tried the following import statements:

// TS2306: '.../@types/googlemaps/index.d.ts' is not a module.
import {google} from "@types/googlemaps";
// Cannot find module @types/googlemaps
import "@types/googlemaps";
// TS2306: '.../@types/googlemaps/index.d.ts' is not a module.
import * as googlemaps from "@types/googlemaps";

It all didn’t work.

After couple hours of fighting with this, solution was found:

  1. Remove any import statement.
  2. Go to tsconfig.json file in your app and add the following records into “compilerOptions” section:
compilerOptions: {
  ...
  "typeRoots": ["./node_modules/@types"],
  "types": ["googlemaps"]
}

You are welcome!

P.S. While writing this article another solution was found on Stack Overflow:

import {} from '@types/googlemaps';

And there is no need to deal with tsconfig.json file.

World GeoJson repository release v2.0.0: areas boundaries

I’m happy to announce that second release of my world-geojson repository is finally online. First release was about countries, and countries there included all the areas mentioned above. But in most cases when you say United Kingdom, you don’t think about Montserrat or South Sandwich islands, although they belong to the Crown. That’s what this release is about:

  • countries being associated with or managed by another countries
  • overseas territories
  • regions not being part of a country mainland

Full list of the areas introduced in the release is in the bottom of this post.

I want to mention that it is quite tricky to choose which countries to split to areas, and which not to. So while doing that I used Google Places API to verify if the area is recognized. For example French Polynesia is recognized, so it was separated from France mainland. But if we take a look on Malaysia, we see that it’s split between two major islands. Informally they are called as “West Malaysia” and “East Malaysia”. However, Google Places don’t recognize such a query, so Malaysia remains as is in this release. Finally you might find it handy that all the areas are compatible with Google Places API.

Also second release contains several fixes for the countries boundaries too. Added US Virgin Islands, improved UK boundaries, added Mayotte and Reunion, etc.

Next release would be about countries states. That would be big, as Russia alone has about 90 states, and in addition to that there are US, China, India, Brasil, Indonesia and other countries. Don’t expect next release soon, but it will definitely come one day.

If you want to contribute in any way – advice, fix, additions – welcome to comments.

Full list of areas introduced in release 2.0.0:

  • Denmark (3): Mainland/Greenland/Faroe Islands
  • Ecuador (2): Mainland/Galápagos Islands
  • France (18): Mainland/Corsica/French Guiana/Guadeloupe/Martinique/Mayotte/Réunion/Clipperton Island/French Polynesia/New Caledonia/Saint Barthélemy/Saint Martin/Saint Pierre and Miquelon/Wallis and Futuna/French Southern and Atlantic Lands
  • Italy (3): Mainland/Sardinia/Sicily
  • Netherlands (7): Mainland/Bonaire/Saba/Sint Eustatius/Aruba/Curaçao/Sin Maarten
  • New Zealand (12): Mainland/North Island/South Island/Tokelau/Kermadec Islands/Chatham Islands/Three Kings Islands/Antipodes Islands/Auckland Islands/Bounty Islands/Campbell Island/Snares Islands
  • Norway (3): Mainland/Svalbard/Jan Mayen
  • Portugal (3): Mainland/Azores/Madeira
  • Spain (3): Mailand/Canary Islands/Balearic Islands
  • United Kingdom (21): United Kingdom/England/Scotland/Wales/Northern Ireland/Isle of Man/Guernsey/Jersey/Gibraltar/Akrotiri and Dhekelia/Bermuda/Turks and Caicos Islands/British Virgin Islands/Anguilla/Cayman Islands/Montserrat/Pitcairn Islands/Saint Helena, Ascension and Tristan de Cunha/British Indian Ocean Territory/Falkland Islands/South Georgia and the South Sandwich Islands
  • USA (17): Mainland (to be splitted to states)/Palmyra Atoll/Guam/Northern Mariana Islands/Puerto Rico/United States Virgin Islands/American Samoa/Baker Island/Howland Island/Jarvis Island/Johnston Atoll/Kingman Reef/Wake Island/Midway Atoll/Navassa Island/Serranilla Bank/Bajo Nuevo Bank

4 steps to secure Wordpress installation on a VPS server

I’m not a huge fan of Worpdress. It’s creators publish security patches and updates so frequently, so every blog owner should hire a system administrator to take care of it. Not that security patches are bad, bad is the fact that so many of them are needed. However, having been watching over my wife’s blog for three years already, I’ve got some recipes on how to secure it and not have a permanent headache about it’s security.

Note: this instruction is for those who knows Linux command line basics.
Another note: My examples are compatible with Ubuntu 16.04.
One more note: Besides Apache + PHP + MySQL I use Nginx as a reverse proxy. It serves static files and provides some functionality which is to be mentioned in this post.

1. Set up correct directory rights

Wordpress has automatic updates for the core and plugins – it looks like a nice feature, but if it works you are in danger. Here is why – that means that your web-server (server software handling requests and sending responses) has rights to modify files in your blog. It means that if somebody will get access to the web-server (through some of countless Wordpress holes), he will be able to modify blog files injecting there malicious code. The only writeable directory for Wordpress should be uploads.

Here is what I do to secure directories:

# 1. Set my user as an owner, and www-data as a group of all the directories and files:
sudo chown george:www-data /my/wordpress/installation -R

# 2. Set directories mode to 755, and files to 644  (read for all, write for owner):
find /my/wordpress/installation -type d -exec chmod 755 {} \;
> find /my/wordpress/installation -type f -exec chmod 644 {} \;

# 3. Set upload directory and it's subfolders mode to 775, and files to 664 (read for all, write for owner and group):
> chmod 775 /my/wordpress/installation/uploads
> find /my/wordpress/installation/uploads -type d -exec chmod 775 {} \;
> find /my/wordpress/installation/uploads -type f -exec chmod 664 {} \;

2. Deny execution of PHP files in directories not supposed to be having PHP files

One of the most commonly used approaches to hack Wordpress I’ve ever seen was attempt to upload some PHP file to uploads directory and run it from there. Besides having correct directory rights, it wouldn’t hurt to deny Apache (or any other web-server) executing PHP files in some certain directories. Here is an example for Apache virtual host:

<Directory ~ "/my/wordpress/installation/(wp-content/uploads)/">
  <Files "*.php">
    Require all denied
  </Files>
</Directory>

Probably there is a way to do this in .htaccess file to make this solution more portable – I haven’t checked.

3. Set up protection against DDoS and bruteforce attacks.

Wordpress’ weak spots are:

  1. wp-login.php page
  2. xmlrpc.php file
    I’ve experienced lots of DDoS attacks on my installation, and logs always show these two files to be a target.

First thing we would do – add another authentication layer to Wordpress. It would be HTTP authorization for wp-login.php file. I use Apache as a web-server and set authorization right in the virtual host:

<Directory /my/wordpress/installation/>
  # Some directory settings

  <Files "wp-login.php">
    AuthType basic
    AuthName "Please enter your username and password"
    AuthUserFile /etc/.htpasswd
    Require valid-user
  </Files>
<Directory>

File /etc/.htpasswd should be created using htpasswd utility.

Now I slow down those who are trying to DDoS or bruteforce my Wordpress. I don’t know how to limit number of requests to a particular file in Apache, but as I sad earlier – I use Nginx which has “Rate limit” functionality, so I added the following settings to it’s configuration files:

# This goes into nginx config (/etc/nginx.conf in my case) into http section. This line will define a zone which will limit requests to 1 per second, and zone's size is 10 Mb.
limit_req_zone $binary_remote_addr zone=mywordpress:10m rate=1r/s;

# This goes into virtual host under server section. All the requests to xmlrpc.php will be handled according to the zone rules, allowing a burst of 5 requests though.
location /xmlrpc.php {
  limit_req zone=mywordpress burst=5;
  proxy_pass http://127.0.0.1:8080;
  include /etc/nginx/proxy_params;
}

location /wp-login.php {
  limit_req zone=mywordpress burst=5;
  proxy_pass http://127.0.0.1:8080;
  include /etc/nginx/proxy_params;
}

What is wrong with xmlrpc.php file? Nothing except it’s commonly used for DDoS attacks. Easier approach would be to just deny access to it, but it’s not a solution in case you want to use some wordpress client. My solution should be better.

4. Addition: protect your phpmyadmin if you have any

I’ve seen tons of access log records showing attempts to find my phpmyadmin – they knocked to /pma, /phpmyadmin, /admin and many other links. So in order to secure it too, I do the following:

  1. Don’t use /phpmyadmin and any other obvious link for PhpMyAdmin. Better use something you can remember, but not related to “php”, “my” and “admin” words, e. g. /blablabla, /ilovemydatabase, /houston.
  2. Bind HTTP authorization to it too. Example is above.

Of course there is more you can do about your wordpress security and safety. I’m not specializing in server administration, but above are recipes which I have been using for some time and having no problems so far. If you have something to add or improve, I would be happy to see you comment.

P.S.: I don’t work with Wordpress a lot. It’s a coincidence having already two posts about Wordpress in my blog. :)

Update 22.11.2017: I thought that PHP files within wp-includes directory are never called directly, and only being included into main script. That’s not correct, for example TinyMCE does a direct call. Thus wp-includes directory should not be blocked in step (2).

2017   security   wordpress

World GeoJson repository release v1.0.0: countries boundaries

I am proud to announce first release of World GeoJson repository. It contains all the countries boundaries with the best quality available for commercial use so far.

Example of quality. Other repositories do not care about small islands so much, I do.

You can download it from GitHub or use as a Composer package:

"repositories": [
        {
            "type": "git",
            "url": "https://github.com/georgique/world-geojson/"
        }
    ],
    "require": {
        "georgique/world-geojson": "1.*"
    }

This data was created manually using http://geojson.io tool, which uses Mapbox, OCM and OSM maps. So the data would be perfectly displayed on these maps, but it has some minor incompatibilities with Google Maps: northern areas and some tiny islands. I’m going to keep working on these issues and get them fixed.

Next release would be about areas boundaries. Some countries have external territories, e. g. Denmark besides the mainland has Greenland and Faroe Islands. I’m going to split them. Date of the next release is not known yet.

Happy using!

Xdebug remote debug configuration tip

Sometimes I need to debug things not locally, but on our development server. Server configuration is pretty classic: PHP works with Apache behind Nginx. As a client I use PhpStorm IDE. Xdebug was configured properly and checked 50 times at least. Same with PhpStorm. But it didn’t work.

Solution was simple and actually I hit this underwater rock not the first time. It’s important to understand on how Xdebug works – although it’s being placed on server, it actually acts like a client. A server would be an IDE. When Xdebug is triggered, it gets your IP address (either hardcoded in config or passed in headers – make sure to check this when using reverse proxy!) and port and connects to it. That means that you can set everything up perfectly, but if your IDE is not reachable from outside, you won’t get your debugger working.

First thing to check would be whether your server has allowed outbound connection to your DBGP port (by default 9000). That’s remote side (I’m trying to prevent client-server mess as above I stated XDebug as a client and IDE as a server). Local side – and that was my problem – is that we usually work behind a router, which masks my computer for an outside world. Yes, already know the answer – DBGP port has to be forwarded on a router to the local IP address.

It seems quite simple now, but with every new configuration I often forget about it and spend hours trying to understand what is going on. Hope this tip would be helpful.

P.S. For those working in some public places (cafes, co-working spaces, etc) – I’m not sure how to help you. Please share your recipes in comments.

P.P.S. For teams working in offices – check out on DBGP proxy, PhpStorm supports it.

2017   debugging   network   php   xdebug

Importing cyrillic Wordpress comments into Disqus

My wife has a blog on Wordpress. She writes in Russian – it’s important to mention (and it’s in the subject). Some time ago I decided to improve the blog a bit, and one of the improvements was moving comments to Disqus. I like Disqus because it makes commenting easier which potentially leads to more comments and discussions. And comments and discussions are good.

I installed Disqus plugin and started to export comments into Disqus. Honestly saying I don’t remember how it ended (it was a while ago), but when I went to Disqus, I found no comments there.

Then I started to try to move comments semi-manually. It means exporting them from Wordpress into .xml file and importing the file into Disqus. It started well – I exported a file successfully. But when I tried to import it into Disqus, page just got reloaded with no message about what is going on. Comments did not appear.

I started to investigate into the issue and found http://import.disqus.com which was a bit more informative on the problem. The error was the following:

XML syntax error: Input is not proper UTF-8, indicate encoding !
Bytes: 0xD0 0xBE 0xD1 0x81, line 51, column 294 (line 51)

If encoding would not be specified and that would be a problem, I would not be writing this post. Surely encoding was indicated. I played with the file really lot -- I was checking it with a hex editor, validating it with XML validators – it all was helpless. I was pretty sure that the problem is connected to cyrillic characters, in fact it might have been, but not directly.

After a long googling, working with the XML file, drinking a lot of tea (I’m a tea drinker, yes) and saying a bunch of “good” words addressed to Disqus importer, I’ve found out a way to get through the issue.

1. Wipe out posts and pages content from the XML

You import comments, not the posts and pages themselves. More contents – more symbols causing troubles. Wipe contents and excerpts, you can even use a regular expression to replace everything between .

<!-- Was: -->
<content:encoded><![CDATA[Привет, тут какой-то текст!]]></content:encoded>
<excerpt:encoded><![CDATA[Тут тоже текст!]]></excerpt:encoded>
<!-- Is: -->
<content:encoded><![CDATA[]]></content:encoded>
<excerpt:encoded><![CDATA[]]></excerpt:encoded>

2. Wipe out all the suspicious punctuation

Regulal expressions would be helpful for this too – I’ve written one to search for everything except commonly used words, digits and punctuation characters. It found me some weird ellipsis, strange dashes and crazy spaces. Removed all of them. You do the same.

3. Do something with the same meaning as shaman dances and hope it helps

Now when you are pretty sure that your file is clean of garbage, it might be imported successfully. Or might be not. Mine was still causing an error, but on different line and column. When I was looking to it, it turned that problem is always caused exactly between CDATA[ opening bracket and first letter of the Cyrillic text. Here is what I did – I added a space there and it worked. When it didn’t helped, I removed the space, tried again. If that didn’t work too – added space again. It’s crazy. I know. Ask Disqus, why. After maximum of three attempts, error was gone and another one appeared until I went through all of them (I had about 150 comments and about 10 errors on this stage).

Finally I was able to import the file. Hope my recipe would be helpful.

P.S. Another small tip which was indirectly mentioned above – don’t forget to move page comments, not posts only. I forgot. Was punched for that. :)

P.P.S. Couple days ago I was importing pages comments which I forgot to import first time. During the import I broke Disqus importer so it started to give me 500 errors. Next day they emailed me saying that there was a bug on their side and they have fixed it. Haven’t checked, but good reaction!

Why I don’t like PHP: inconsistency

There are tons of programming languages nowadays. Some of them are low-level, some of them high-level. Some of them have more features and so-called syntax-sugar, some of them – less. Some of them have more extensions and libraries, some of them – less. Some of them are easier to learn, some of them harder. What I want to say is that it doesn’t really matter. The most pragmatic approach is to use the one which you are going to be paid for.

However, good language makes your life easier. Besides all the things said above, it should be intuitively understandable. You might not know how to write some things, but with a good language you would have an idea. This is what in my opinion PHP is lack of.

Consistency is one of the key pillars of intuitive understanding. If you know that string functions always start with str_ prefix, and you are looking into some new function, you would just type str_ in your IDE and wait for autocompletion tips. Seriously, if a function is named well, you would find it. But not in PHP. In PHP string functions start with str_, str or often have no prefix.

Function replacing a substring into something else is str_replace(). Functions trimming garbage are trim(), ltrim() and rtrim(). If you want to know string length, you go with strlen() function. You can’t keep this in mind, you can’t intuitively find needed function, so often you just have to google it.

I remember once my boss told me that the better interface of an application is, the less calls he will have to the support, which leads to less operational costs. From this point of view this PHP naming hell is very unfortunate.

Now let’s take a look on array functions. Naming is better there, most of the functions start with array_ prefix. But don’t relax, it’s not over yet. Let’s say we have an array of some objects and we need to have an array of their IDs. We go with array_map() function:

$objects = array($var1, $var2, ..., $varN);
$ids = array_map(function ($obj) { return $obj->id; }, $objects);

Pretty clear, isn’t it?. Function array_map() receives first parameter as a callback function, second as an array. Now when we have an array of IDs, let’s say we want to filter it and return only those IDs which are less than 10 (I know, it is quite stupid, can’t imagine why I would need to do this, but whatever):

// Now knowing array_map() function you write the following:
$filtered_ids = array_filter(function($id) { return $id >= 10; }, $ids);

Aaaaaand... this will not work! Because array_filter() receives an array as the first parameter, callback should be the second one:

$filtered_ids = array_filter($ids, function($id) { return $id >= 10; });

What the hell? How such inconsistencies would happen?

Toolbox: PHP sandbox and Regex tester

I’ve heard a lot of people using JS sandbox JSFiddle although I’m not much of a frontend developer. But never heard about somebody using any PHP sandbox.

Here I am, and here is PHP sandbox I love to use. It has a bunch of different PHP versions and a convenient code formatter which makes it a great everyday tool.

That’s it. Short post.

In order to make it longer I would share a link to regular expression tester which I also love a lot. I don’t know if anybody know about it, but have never heard a mention about it. This tool is so good that I used it first to understand what the heck are the regular expressions, and then learned there how to write them. Can any other tester or debugger do that?

Which instruments do you use? Share them in comments.

Picture from https://advivalares.wordpress.com/2013/05/14/criancas-ganharao-bercario/

Transferring Amazon EC2 instance to another region

Recently we have launched an AWS EC2 instance in Canada Central region and created a S3 storage to use with it. We couldn’t connect to the storage and while investigation it appeared that AWS SDK for PHP v2 seems to be not compatible with this region – it worked well with US East region well though.

So we decided to move both instance and storage to US East region. With S3 it was pretty easy for us – we had no data there yet, so just dropped the bucket and created another one in US East.

With EC2 instance it’s a bit more tricky – we had a configured server with installed software and we didn’t want to reinstall everything on another instance, so we needed a transfer. Transfer of the instance itself is not possible, but you can transfer AMI – Amazon Machine Image.

An Amazon Machine Image (AMI) provides the information required to launch an instance, which is a virtual server in the cloud. You specify an AMI when you launch an instance, and you can launch as many instances from the AMI as you need. You can also launch instances from as many different AMIs as you need.

Seems pretty clear. Now here are quite simple steps on how to do the transfer:

  1. Stop an instance in order to ensure data integrity there – you don’t want create an image in the middle of some operation which might lead to data corruption, and risk of that is quite high if the instance is working.
  2. Choose the instance and go to menu Actions -> Image -> Create Image. Give it some name, ensure all the instance’s volumes are there and create AMI.
  3. After some time go to the list of your AMIs and you will see a freshly created one. Choose it and go to menu Actions -> Copy AMI. Choose destination region and start copying. Don’t forget to switch region in the console in order to find destination AMI.
  4. When AMI in destination region becomes available, choose it and go to menu Actions -> Launch. Then just launch an instance as usual – you will have to set up all instance options as for the blank new instance. Note that you will have to define security group there again, also Elastic IP can’t be transferred, so in case of using it on the old instance, you’ll just have to allocate a new IP for the new instance.

That’s it! When you connect to the instance, you will find out all the software and data is there. Don’t forget to do a clean up – AMI, old instance, old volume, etc, also change IP addresses pointing to the old instance, e. g. DNS records.

You can read AWS documentation on AMI transfer here.

Picture from https://tproger.ru/translations/aws-in-plain-russian/

2017   aws   ec2
Ctrl + ↓ Earlier