You can see the code here, too: https://gist.github.com/axelei/dafd305213ed1b2655545f06edfbc905

# Programa para clonar prestashop
# Copyright Krusher 2015 - Licenciado bajo GPL3

paramfile="httpdocs/app/config/parameters.php"
extractDatabaseName="s/.*'database_name' => '\(.*\)'.*/\1/p"
extractDatabaseUser="s/.*'database_user' => '\(.*\)'.*/\1/p"
extractDatabasePassword="s/.*'database_password' => '\(.*\)'.*/\1/p"

if [ "$#" -lt 4 ]; then
  echo "Error: Se requieren 3 argumentos."
  echo "Uso: $0 dir_web_original dir_web_destino url_original url_destino"
  exit 1
fi

echo Traspasando de $1 a $2 \(urls: $3 -- $4 \)

databaseName=$(sed -n "$extractDatabaseName" $1/app/config/parameters.php)
databaseUser=$(sed -n "$extractDatabaseUser" $1/app/config/parameters.php)
databasePassword=$(sed -n "$extractDatabasePassword" $1/app/config/parameters.php)

echo Datos de la base de datos de origen: $databaseName / $databaseUser / $databasePassword

targetName=$(sed -n "$extractDatabaseName" $2/app/config/parameters.php)
targetUser=$(sed -n "$extractDatabaseUser" $2/app/config/parameters.php)
targetPassword=$(sed -n "$extractDatabasePassword" $2/app/config/parameters.php)

echo Datos de la base de datos de destino: $targetName / $targetUser / $targetPassword

if [[ -z "$databaseName" ]] || [[ -z "$databaseUser" ]] || [[ -z "$databasePassword" ]] || [[ -z "$targetName" ]] || [[ -z "$targetUser" ]] || [[ -z "$targetPassword" ]]; then
  echo "Error: Algunas de las variables no han sido extraídas, comprueba los errores y los directorios."
  exit 1
fi

echo URL de la instancia origen: $3
echo URL de la instancia destino: $4

read -p "Pulsa enter para continuar o CTRL+C para cancelar"
echo  "Procediendo con el traspaso."

echo Extrayendo base de datos
rm -f temp.sql
mysqldump -u $databaseUser $databaseName -p$databasePassword > temp.sql
echo Borrando base de datos objetivo
mysql -u $targetUser $targetName -p$targetPassword -e "drop database $targetName; create database $targetName"
echo Insertando la base de datos origen en la objetivo
mysql -u $targetUser -D $targetName -p$targetPassword < temp.sql
rm -f temp.sql

echo borra ficheros contenido antiguo
rm -Rf $2/*
echo copia ficheros
cp -r $1/* $2/

echo Configurando base de datos objetivo con los datos originales

sed -i "s/'database_name' => '\(.*\)'/'database_name' => '$targetName'/g" $2/app/config/parameters.php
sed -i "s/'database_user' => '\(.*\)'/'database_user' => '$targetUser'/g" $2/app/config/parameters.php
sed -i "s/'database_password' => '\(.*\)'/'database_password' => '$targetPassword'/g" $2/app/config/parameters.php

echo Configurando la nueva URL en la instancia destino

mysql -u $targetUser $targetName -p$targetPassword -e "update prstshp_configuration set value = '$4' where NAME IN ('PS_SHOP_DOMAIN', 'PS_SHOP_DOMAIN_SSL')"
mysql -u $targetUser $targetName -p$targetPassword -e "UPDATE prstshp_shop_url SET domain = '$4', domain_ssl = '$4' WHERE id_shop_url = 1;"
sed -i s/\\^$3\\$/^$4$/g $2/.htaccess

echo Terminado.

The implementation could undoubtedly be improved—using a more suitable language or more advanced tools—but this is the solution I came up with given the constraints of an environment lacking many essential utilities and commands. If you found it helpful, feel free to share your experience in the comments.

La entrada About how to clone a Prestashop instance se publicó primero en Nobody comes after the last.

There were ports in a great number of platforms, from the Macintosh (the original) to GameBoy Advance, but my favourite is MS-DOS, and it’s what this article is about. There are two interesting files: the executable (SC2000.EXE) and the data file (SC2000.DAT). Unfortunately, Windows version didn’t came out in Spanish (my mother language), and Network Edition version works awfully bad (and it is available only in English).

SC2000.EXE

The game executable does not seem to have many resources, but it does have some interface texts. In the hexadecimal editor we can see some fixed-width labels.

There are some texts representing variable-width labels, too. There are also some embedded files, in which for example game scenarios are described.

At the moment disassembling an executable from the early 90s is not one of my specialities, and thus I have not find out much. Pointers are not evident in the executable, so I left it alone. But the most interesting is the data files we are describing down below.

SC2000.DAT

This is the main data file. It does not have a header, but a short look in the hexadecimal editor throw some hints about its structure.

It turns out that from byte 0 the first we can find is 16-byte blocks that describe files contained in the pack. The first field is evident: 12 bytes with the file name (8 + 3 characters of the MS-DOS file with the period). If there is leftover space, the rest of the bytes are 00h.

The other two bytes are not so evident, but it turns out that the game is originary from Macintosh, which at that time used Motorola processors. These processors, unlike Intel‘s, are Little-Endian (being Intel Big-Indian). This means that numbers with more than one byte are stored ordered from the least significant byte to the most significant, instead of the “natural” ordering. This is thus a 32 bit unsigned integer in little-endian format, which codifies the offset of the file just before.

I owe the happy idea about the offset to Brett Lajzer, an Albany software engineer that began researching this file before me. I wrote him to swap information and advised me about this point, which he didn’t finally describe in his article.

Unpacking and packing, knowing this, is relatively simple. I’ve written a small Java program that makes this operation easy:

package sce2000;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.Serializable;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;

import org.apache.commons.io.FileUtils;

public class Sce2000 {
	
	public static final String SC2000DAT = "G:\\dos\\sce2000\\sc2000.dat";
	public static final int NUMFILES = 399;
	
	public static class Filestrut implements Serializable {
		private static final long serialVersionUID = 1L;
		public String filename;
		public int offset;
		public int targetOffset;
	}

	public static void main(String[] args) throws IOException, ClassNotFoundException {
		
		if (args.length == 0) {
			info();
		} else if ("x".equals(args[0])) {
			extract();
		} else if ("c".equals(args[0])) {
			create();
		} else {
			info();
		}

	}
	
	public static void info() {
		System.out.println("Usage: x to eXtract or c to Create (after eXtract) + sc2000.dat file");
	}

	public static void create() throws IOException, ClassNotFoundException {
		
		File sc2000dat = new File(SC2000DAT);
		File metafile = new File(SC2000DAT + "!/meta");
		ObjectInputStream ois = new ObjectInputStream(new FileInputStream(metafile));
		List<Filestrut> files = (List<Filestrut>) ois.readObject();
		
		List<Byte> targetFile = new ArrayList<Byte>();

		for (Filestrut file : files) {
			byte[] fileNameBytes = file.filename.getBytes(StandardCharsets.US_ASCII);
			for (Byte myByte : fileNameBytes) {
				targetFile.add(myByte);
			}
			for (int i = 0; i < 16 - fileNameBytes.length; i++) {
				targetFile.add((byte) 0);
			}
			//sourceFile.length();
		}
		
		int i = 0;
		for (Filestrut file : files) {
			byte[] binary = Files.readAllBytes(Paths.get(SC2000DAT + "!/" + file.filename));
			int fileOffset = targetFile.size();
			for (Byte myByte : binary) {
				targetFile.add(myByte);
			}
			int filePointer = 12 + (16 * i);
			
			byte[] offsetBytes = fromInt(fileOffset);
			targetFile.set(filePointer, offsetBytes[0]);
			targetFile.set(filePointer + 1, offsetBytes[1]);
			targetFile.set(filePointer + 2, offsetBytes[2]);
			targetFile.set(filePointer + 3, offsetBytes[3]);
			
			i++;
		}
		
		byte[] binaryFile = new byte[targetFile.size()];
		for (i = 0; i < targetFile.size(); i++) {
			binaryFile[i] = targetFile.get(i);
		}
		
		FileUtils.writeByteArrayToFile(sc2000dat, binaryFile);

		
		System.out.println();
		
	}
		

	public static void extract() throws IOException {
		
		Path sc2000dat = Paths.get(SC2000DAT);
		byte[] data = Files.readAllBytes(sc2000dat);
		
		int vector = 0;
		List<Filestrut> files = new ArrayList<Filestrut>();
		for (int i = 0; i < NUMFILES; i++) {
			
			Filestrut currfile = new Filestrut();
			
			currfile.filename =  convertFilename(Arrays.copyOfRange(data, vector, vector + 12));
			vector += 12;
			
			currfile.offset = fromByteArray(Arrays.copyOfRange(data, vector, vector + 4));
			vector += 4;
			
			System.out.println("Found file " + currfile.filename + " at " + currfile.offset);
			
			files.add(currfile);
			
		}
		
		File dir = new File(SC2000DAT + "!");
		if (dir.exists()) {
			FileUtils.deleteDirectory(dir);
		}
		
		dir.mkdir();
		
		Iterator<Filestrut> fileIt = files.iterator();
		
		Filestrut file = fileIt.next();
		Filestrut fileNext = null;
		
		boolean stop = false;
		while (!stop) {
			
			File extracted = new File(SC2000DAT + "!/" + file.filename);
			
			int init = file.offset;
			int end = -1;
			if (fileIt.hasNext()) {
				fileNext = fileIt.next();
				end = fileNext.offset;
			} else {
				end = data.length;
				stop = true;
			}

			System.out.println("Writing: " + extracted.getAbsolutePath());
			FileUtils.writeByteArrayToFile(extracted, Arrays.copyOfRange(data, init, end));
			
			file = fileNext;
			
		}
		
		File metafile = new File(SC2000DAT + "!/meta");
		
		ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(metafile));
		oos.writeObject(files);
		oos.close();
		
		System.out.println("OK!");
	}
	
	public static byte[] fromInt(int number) {
		byte[] bytes = ByteBuffer.allocate(4).putInt(number).array();
		swapEndianess(bytes);
		return bytes;
	}
	
	public static void swapEndianess(byte [] bytes) {
		byte temp1 = bytes[0];
		byte temp2 = bytes[1];
		bytes[0] = bytes[3];
		bytes[1] = bytes[2];
		bytes[2] = temp2;
		bytes[3] = temp1;
	}
	
	public static int fromByteArray(byte[] bytes) {
		// Change endianness
		swapEndianess(bytes);
		return ByteBuffer.wrap(bytes).getInt();
	}
	
	public static String convertFilename(byte[] data) {
	    StringBuilder sb = new StringBuilder(data.length);
	    for (int i = 0; i < data.length; ++ i) {
	        if (data[i] < 0) { 
	        	throw new IllegalArgumentException();
	        }
	        if (data[i] == 0) {
	        	break;
	        }
	        sb.append((char) data[i]);
	    }
	    return sb.toString();
	}


}

This program has been done quick and dirty and it’s not what we would say optimized. I indulged myself loading the whole files in memory as the complete DAT file is few megabytes big. Also it has the file count and DAT path hardcoded in the code. Making it better and more efficient is left as an exercise to the reader.

Why in Java? It’s not the most appropriate for handling binaries, and not having unsigned types support doesn’t precisely help. I simply did it in Java and not in C because it’s the language I use for a living, and the one I’m the most fluent with. Also I’ve not programmed anything in C for 10 years. 🙂

Using the above code (or any other the intrepid reader can code) we can see the following file types:

• RAW, as the header-less image files.
• PAL, as the colour palette used in the game.
• Text files, some as TXT* without an extension and others with RAW extension.
• XMI, with the music.
• VOC, with the sound effect files.
• FNT, with the typography files.
• Etc, etc…

Let’s talk about them a bit one by one.

VOC files

The game sound files are stored in VOC files. Well, that’s one is easy, as it’s not a common file format but broadly supported on many sound edition programs. It’s a format by Creative Labs that chiefly stores PCM and ADPCM coded audio, although sometimes has been used for other encodings.

It can be perfectly open and saved with Adobe Audition. The sampling frequency varies from one file to another, but is recognized without many problems.

Text files

There are three kinds of text file types. The most simple are TXT* files without an extension, being * a number. They can be directly opened with Notepad++ and be edited without problems.

Well, not quite. Encoding is not ASCII, nor UNICODE because it wasn’t common at the time. Which one, then? Well, being the game was programmed for Macintosh, encoding is Macintosh Roman. This is a problem because Notepad++ does not support it. It’s been asked for, but seems not the priority, so you might want to convert it somehow to edit it more comfortably.

Addendum: It supports it after all! The option was buried in the menus: Encoding > Character Sets > Cyrillic > Macintosh.

Other text files are STR*.RAW and *.RAW, being * a number. Their format is not so nice.

So the first byte is always 00h, and the second seems to store the string count in the text. Strings, unlike in C, aren’t coded terminated in 00h. They are stored preceeded by its length, coded in 1 byte. Ideally we could use an utility software for this, which could easily be coded. (but I don’t feel like to)

The third text file kind correspond only to PPDT1003.RAW file, which has one of the funniest characteristics in all SimCity 2000: the newspapers.

Unfortunately this format is a nightmare. It begins with a term dictionary separated by 00h, and then a clump of texts used to generate the articles. This custom encoding is variable-width (for an example, 5C96h represent the ñ character) and uses the previously defined terms, and of course placeholders for article subjects. These articles are procedurally generated, and an article denouncing the disappearance of an animal had as the main protagonist a cat or a rhinoceros, owned by mrs. Dwight or mr. Martínez.

Back in the day we had editors like Thingy, which supported TBL files. You could specify a term dictionary there and easily edit files like this. Anyway, it would only be of use if we could keep the string length, and even then we would need to decipher the preceding number (and its encoding) so we could properly edit it.

Addendum: It seems the pointers for the newspaper texts are in the PPDT1004.RAW file. They consist in blocks of 4 bytes in big-endian. What a mess.

Image files

Excluding some text files (described in above), RAW files store the images used in the game interface. Buildings and other elements are in some DAT files that would be out of the scope of this article, because they can be edited much more easily with the SimCity Urban Renewal Kit (SCURK), a tool included in some later versions which could edit cities “by hand” and modify the graphical aspect of the buildings.

Now about the files in question, these RAW files are 8 bit (256 colours) bitmaps with a 4 byte header. It doesn’t seem to be relevant information about resolution or colours, but for that matter we have another file, MINE.PAL, which specifies the colour palette used throughout the game. It’s not a Microsoft Palette file as the extension suggest, but a raw palette ACT type. Interestingly, it’s the preferred format used in Adobe Photoshop.

So for the resolution, there seems not to be an easy way to find it out simply looking at the file. The most easy method is factorizing the number (minus 4 bytes) in two factors, something Photoshop could help us with when opening the file, at least in the most modern version. By the file name we can more or less tell what’s the file for, and in case we have played (you really should!) the game, you could more or less recognize its width/height ratio. Some are easy: the title screen (TITLE.RAW) is a 640×480 image. All the images use the same palette, as the game only uses the same 256 colours all of the time.

I always use the same technique editing 8 bit indexed colour images: I convert them to 24 bit colour (plus 8 bit alpha channel) and I edit them comfortably. After that, before saving them, I change them to indexed color loading again the game palette. Surprisingly enough, saving the files in Adobe RAW format, the game loads it right, because Photoshop would allow us to preserve the 4 byte header. What a relief!

Please note: needless to say this RAW format has anything to do with raw image formats the nowadays digital cameras uses.

XMI files

Game music tracks are stored in the XMI files. Tis format supports many tracks per file, but this seems not the case. The structure is very different from MIDI files, but they do about the same: store musical notes and events. It can be played in Windows easily with Foobar2000, and it is possible to convert MIDI to XML (and vice versa). MIDIPLEX seems to be able to do it from Windows, but no compiled binaries are offered so I did it with older tools with MS-DOS, using DOSBox. Anyway, the very SimCity 2000 will need DOSBox to work in modern systems.

FNT files

They are presumed to be the game typographies, to judge from the numbers and the extension they come with. It seems not the same format that Windows uses (bitmap FNT), so I’ve been unable to open the files with any editor.

Others

Other files present in the package are General MIDI sources for OPL chipsets, some indexes and headers for the building graphic sets… anything very interesting when modifying the game. Everything else seems to be embedded in the executable file, something it’s out of my reach at the moment.

In Windows

On a point of information, the Windows version can be modified much more easily. The image and sound resources are present in WAV and BMP format, which are much less obscure as the previously described. The rest of the resources are embedded in the game executable, but with a resource editor they can be extracted and modified with ease, and in more accessible format than their MS-DOS counterparts. (I use Resource Hacker, which is free and works quite well)

Newspapers, however, seems to have the same calamitous format than the MS-DOS version. It’s a pity, because it would be very interesting to translate this version to Spanish.

So this is where I leave it. I hope it’s been educative, although after all this time there seems to be little interest modifying this game. I’ve always wanted to meddle with this game and create my own mod, Sim City 2000 effect, a bit mischievous, but maybe that’ll be in another time.

La entrada Taking Sim City 2000 into pieces se publicó primero en Nobody comes after the last.

Then how can we link a HTTP control to an object? For that, JSF provides converters. Well, it will allow you to program them, of course. Neither PrimeFaces, my library of choice, includes these converters. Thus, I’ve programmed a simple one for SelectOneMenu, the dropdown menus in the like of a combobox. I’ll leave it here for my own reference, and for anyone that could use it.

/**
 * The Class SelectOneMenuConverter.
 */
@FacesConverter("selectOneMenuConverter")
public class SelectOneMenuConverter implements Converter {

	@Override
	public Object getAsObject(final FacesContext arg0, final UIComponent arg1, final String objectString) {
		if (objectString == null) {
			return null;
		}

		return fromSelect(arg1, objectString);
	}

	/**
	 * Serialize.
	 *
	 * @param object
	 *            the object
	 * @return the string
	 */
	private String serialize(final Object object) {
		if (object == null) {
			return null;
		}
		return object.getClass() + "@" + object.hashCode();
	}

	/**
	 * From select.
	 *
	 * @param currentcomponent
	 *            the currentcomponent
	 * @param objectString
	 *            the object string
	 * @return the object
	 */
	private Object fromSelect(final UIComponent currentcomponent, final String objectString) {

		if (currentcomponent.getClass() == UISelectItem.class) {
			final UISelectItem item = (UISelectItem) currentcomponent;
			final Object value = item.getValue();
			if (objectString.equals(serialize(value))) {
				return value;
			}
		}

		if (currentcomponent.getClass() == UISelectItems.class) {
			final UISelectItems items = (UISelectItems) currentcomponent;
			final List<Object> elements = (List<Object>) items.getValue();
			for (final Object element : elements) {
				if (objectString.equals(serialize(element))) {
					return element;
				}
			}
		}


		if (!currentcomponent.getChildren().isEmpty()) {
			for (final UIComponent component : currentcomponent.getChildren()) {
				final Object result = fromSelect(component, objectString);
				if (result != null) {
					return result;
				}
			}
		}
		return null;
	}

	@Override
	public String getAsString(final FacesContext arg0, final UIComponent arg1, final Object object) {
		return serialize(object);
	}

}

Basically, it converts any objects in a coded string with the class name and its hash. When you want to recover it, it access the collection linked to our dropdown menu value to look up the object so it can return it. It relies on the proper implementation of the hashCode method, and it requires it to return different values for different objects, but it’s a nice and quick way to do it, and will work in almost all circumstances. Using it is as easy as:

<p:selectOneMenu id="utiSelector" converter="selectOneMenuConverter" value="#{myBean.myObject}">	
        <f:selectItem itemValue="#{null}" noSelectionOption="true" />
	<f:selectItems value="#{myBean.objectList}" var="object" itemValue="#{object}" itemLabel="#{objeto.label}">
</p:selectOneMenu>

Something I usually do in the DTOs or entities is to make the hashcode to be the key, or the key’s hashcode if it’s compound or non numeric. But that’s another story.

La entrada JSF converter for SelectOneMenu se publicó primero en Nobody comes after the last.