Database Users

Swap performance - Swap Partition or Swap File?

Hi All,
I have been trying to find the best way to configure my swap for my android phone, currently the G1. I had started out with the regular swap partition but find my phone performance degraded over time thus I decided to test the usual read/write for my Class 6 Micro SDHC card.
Configure the SD card via fdisk and setup 3 partitions (like everyone else).
Command (m for help): p
Disk /dev/sdm: 8166 MB, 8166309888 bytes
224 heads, 56 sectors/track, 1271 cylinders
Units = cylinders of 12544 * 512 = 6422528 bytes
Disk identifier: 0x3114df25
Device Boot Start End Blocks Id System
/dev/sdm1 1 800 5017572 c W95 FAT32 (LBA)
/dev/sdm2 801 1100 1881600 83 Linux
/dev/sdm3 1101 1271 1072512 82 Linux swap / Solaris
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: If you have created or modified any DOS 6.x
partitions, please see the fdisk manual page for additional
information.
Syncing disks.
Afterward, format the drive as vfat, ext3, swap
localhost ~ # mkfs.vfat /dev/sdm1
mkfs.vfat 3.0.1 (23 Nov 2008)
localhost ~ # mkfs.ext3 /dev/sdm2
mke2fs 1.41.3 (12-Oct-2008)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
117600 inodes, 470400 blocks
23520 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=482344960
15 block groups
32768 blocks per group, 32768 fragments per group
7840 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 26 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
localhost ~ # mkswap /dev/sdm3
Setting up swapspace version 1, size = 1072508 KiB
no label, UUID=2046ef6b-ac5a-47a2-9484-ea2a356cc7fa
now the read performance via "hdparm -t /dev/sdxxx"
sdm1 (fat) = Timing buffered disk reads: 26 MB in 3.12 seconds = 8.34 MB/sec
sdm2 (ext3) = Timing buffered disk reads: 26 MB in 3.11 seconds = 8.35 MB/sec
sdm3 (swap) = Timing buffered disk reads: 26 MB in 3.08 seconds = 8.45 MB/sec
as you can see, reading is more or less the same across the board.
now the fun part of write. for fat and ext3, I decided to write to a file, for the swap partition, I decided to write directly to the partition.
FAT:
localhost ~ # dd count=30 bs=1M if=/dev/urandom of=sdm1/test.write
30+0 records in
30+0 records out
31457280 bytes (31 MB) copied, 4.9464 s, 6.4 MB/s
EXT3:
localhost ~ # dd count=30 bs=1M if=/dev/urandom of=sdm2/test.write
30+0 records in
30+0 records out
31457280 bytes (31 MB) copied, 4.94644 s, 6.4 MB/s
SWAP:
localhost ~ # dd count=30 bs=1M if=/dev/urandom of=/dev/sdm3
30+0 records in
30+0 records out
31457280 bytes (31 MB) copied, 8.79714 s, 3.6 MB/s
As you can see, write to a FAT or EXT3 partition seems to be faster than writing to a swap partition. Of course, this is assuming I have done my test correctly . If writing to FAT or EXT3 is faster and reads are the same, would it mean that you get better SWAP performance via a swap file on an EXT3 partition?

thanx for the interesting read

Your write test is flawed.
Flaw 1: You use /dev/urandom as a data source. Data generation from /dev/urandom is heavily affected by CPU usage AND by available entropy -- though it doesn't block, its speed does vary drastically.
Flaw 2: You only run the test ONCE. This means that issues from flaw 1 are totally visible rather than being averaged out over a large number of tests. You should do the test 100's of times and take the average over them.
Flaw 3: Implementation of the ext3 filesystem tend to hide the effects of disk latency. There are caches and journals that you haven't accounted for. In order to get the ACTUAL time that it takes to write to the ext3 filesystem, you need to follow the write test by a synchronization, which will *actually* write the data to the disk (i.e., it will BLOCK until the write is actually completed). Unfortunately though, this test will also synchronize OTHER write operations that are buffered, which will further skew your results. What you REALLY need to do is perform the tests on a completely empty partition. Should be starting sync, begin time, test write, finishing sync, finishing time. Elapsed time = finishing time - begin time.
In other words, I'm sorry, but your conclusions are not correct. What I *very strongly* suspect, is that if you do your tests correctly, you will find that the performance of swap-on-ext3 will actually be *lower* than swap-on-partition due to the extra overhead of the ext3 filesystem.

Note: the linux vfat filesystem implementation also buffers writes.
A little bit of fun trivia:
Back in the days when we actually used floppy disks, you could actually copy an entire disk worth of data over to a vfat floppy disk without it even BEGINNING to write to the disk. Usually, it would begin writing several seconds after the copy "completed". If you wanted to speed it along, a sync or umount would force the buffer immediately to write to disk.

ahhh...thanks for the info. I guess with buffer, it makes timing or calculating the write speed a bit difficult. I will have to think more about this.
This really makes me wonder how do SD card manufactures test their write speed and assign the card class.

I believe they do a test similar to yours without any buffering and write different amounts of data to simple file systems.
Then, according to http://en.wikipedia.org/wiki/Secure_Digital the class number is equal to the amount of minimum write megabytes per second that can be written to the disk when in a clean and unfragmented state. So a class 6 card would at it's slowest writing speed on a perfectly setup disk be able to write data at 6 megabytes per second or faster. Read speed isn't taken into account for the rating, but is generally faster than write speed.

I redid my test with raw write via if=/dev/zero and the write speed is now on-par with what was classified on the SD class. (7.4MBps write on my class 6). Playing around w/ the setting more, seems like the over-time degradation seems to be caused by my swappiness configuration.

Formating Internal Memory

I own a SonyEricsson Xperia X1.
According to SKTools Main Storage is formated this way:
It has sectors with 2048 bytes size and 2048 bytes cluster size. So 1 sector per cluster. Also it is formated in FAT32.
On the other hand Storage Card has 512 bytes sector size and 4096 bytes cluster size. So 8 Sector per cluster. Also FAT32.
512 bytes sector is the norm for HardDiskDrives as well. Optical Disk Drives have 2048 bytes sector (i think).
There is going to be a move to 4096 bytes sector for HDDs starting January 2011, which gives more free space and more speed.
Could we do something like that for our phones? 4096 byte sectors and 4096 bytes clusters?

I guess storage cards are diferent Memory arrangements... we have many tweaks to have bigstorage ROMS, search and read bud

Partition information

Not really big news for anyone who's installed SDE and poked around a bit, but I thought I'd post this anyway.
Taken from a 16GB Archos 101. Your results may differ.
Code:
Disk /dev/mmcblk0: 536 MB, 536870912 bytes
4 heads, 16 sectors/track, 16384 cylinders
Units = cylinders of 64 * 512 = 32768 bytes
Device Boot Start End Blocks Id System
/dev/mmcblk0p1 5 1605 51200+ 83 Linux
/dev/mmcblk0p2 1605 5511 125023 83 Linux
/dev/mmcblk0p3 5512 6489 31296 83 Linux
/dev/mmcblk0p4 6490 16384 316640 83 Linux
Disk /dev/mmcblk1: 15.3 GB, 15388901376 bytes
4 heads, 16 sectors/track, 469632 cylinders
Units = cylinders of 64 * 512 = 32768 bytes
Device Boot Start End Blocks Id System
/dev/mmcblk1p1 1 469632 15028216 c Win95 FAT32 (LBA)
Being MMC rather than MTD devices, it ought to be possible to repartition to shuffle some space around, perhaps gaining as much as 50MB extra space for /data.
Additionally, for custom firmwares maybe even repartitioning mmcblk1 may be possible, to create an alternate partition to mount /data/ to.
Of course, for fear of ruining my 101 I have only used fdisk for read-only operations.

don't forget mmcblk2..
it's the external sdcard

Of course I was examining the internal flash storage of the device.
Having my 101 for just under 3 weeks I'm not quite brave enough to repartition mmcblk0, as it would be very difficult to recover an operational system if this were to not go as well as expected. :>
I'm hoping that someone with a little more insight or perhaps a little braver than I could chime in on the possibility.

why won't you repartition (shrink p1 and append other partitions) blk1? (8 / 16 GB storage)
don't touch blk0 when you don't have to

chulri said:
why won't you repartition (shrink p1 and append other partitions) blk1? (8 / 16 GB storage)
don't touch blk0 when you don't have to
Click to expand...
Click to collapse
This was of course my suggestion for alternate firmwares that can take advantage of this.

Moved to general as not android development

[HOWTO] Fix and prevent severe system freezes without wiping the device

Not so long ago I started experiencing severe freezes and lockups on my I9300. The symptoms were well known as pre-SDS lockups. Before SDS workarounds have been released such symptoms would mean a death sentence for the device. While this is not the case any more the freezes still make the phone virtually unusable.
So far the only way to bring the phone back to usability was a full wipe of the device combined with filling the partitions with some data in order to ensure that every sector has been written to. Some have also succeeded to get rid of such behaviour after flashing stock firmware (after having been running AOSP).
Some technical background
The reason for the freezes (at least on my device) were block read errors. This is roughly equivalent to a situation where you HDD starts having bad sectors, however flash memory is a bit different to a magnetic disk drive. The read errors encountered most likely due to data corruption resulting in CRC errors during read. In my case the dmesg (kernel log) showed the following errors:
Code:
[ 5896.313473] c0 mmc0: it occurs a critical error on eMMC it'll try to recover eMMC to normal state
[ 5896.482283] c0 mif: set_hsic_lpa_states: 304: called(exynos4_check_usb_op+0x84/0x100):
[ 5896.482469] c0 mif: set hsic lpa enter: active state (0), pda active (0)
[ 5896.665712] c0 mmc0: Fixing MoviNAND SDS bug.
[ 5896.708336] c0 mmc0: Verifying SDS patch.
[ 5896.794283] c0 mmc0: recovering eMMC has been done
[ 5896.794412] c0 brq->sbc.opcode=23,brq->cmd.opcode=18.
[ 5896.794539] c0 brq->sbc.error=-131,brq->cmd.error=0, brq->stop.error=0,brq->data.error=0.
[ 5896.794789] c0 mmcblk0: unknown error -131 sending read/write command, card status 0x900
[ 5896.795869] c0 end_request: I/O error, dev mmcblk0, sector 21590248
[ 5896.796034] c0 end_request: I/O error, dev mmcblk0, sector 21590256
[ 5896.796182] c0 end_request: I/O error, dev mmcblk0, sector 21590264
[ 5896.796379] c0 CMD aborting case in MMC's block layer ret 0.
[ 5896.796512] c0 mmcblk0: CMD18, ARG=0x14970e8.
[ 5896.796613] c0 packed CMD type = 0.
[ 5896.796700] c0 mmc0, request returns 4.
Instead of doing a full wipe of the device's internal flash memory partitions I've tried out another method. I was looking for a program that would rewrite all the data without actually wiping it. Such operation would cause the data to be written again and additionally the wear levelling algorithm inside the eMMC controller would have the chance to relocate the data to another physical block on the flash memory chip.
It turned out that the program I was looking for is already there - just not fully thought of being the right one to use in such case.
When bad blocks encounter on a HDD on your PC you normally need to scan the whole media and mark all the bad blocks found on the filesystem so that they can no longer be allocated for file storage. On Linux this task is performed by the e2fsck system utility which uses another utility called badblocks to do the scan and report the unusable blocks. It is the latter that is interesting.
Among several options found in the badblocks utility there are three methods for discovering such broken blocks:
- read-only where the program will attempt to read all blocks one-by-one.
- non-destructive read-write in which case the program will for each block: read it, fill it with a pattern and write back the original content.
- destructive write where all blocks will be filled by pattern(s) destroying any existing content.
The two last modes are interesting for the purpose of rewriting your device's memory partitions to get rid of bad blocks.
The non-destructive read-write mode can be used to attempt to recover without wiping the contents of the partition. This is mostly relevant to the /data partition (and possibly /efs) as all other ones can be easily restored. Note however that when a read error occurs the data in that sector is already lost. When you attempt to recover such partition it will appear to be working again, but some apps may be crashing randomly as some of their data is corrupted. The non-destructive read-write mode can also be used to prevent freezes from happening. All you need is to treat the /data partition this way once a while (not too often - every 6 months without a factory reset).
The destructive write mode can be used in case you don't care for the data. This is equivalent to the methods used so far (wiping the partition and filling it with random junk). This mode can also be used to securely* wipe your phone before selling it or giving it away.
* Due to wear levelling this is actually not a secure wipe as some of the data will still persist on the Flash memory chip. However it is secure from a user point of view as the data is no longer visible and recovering it will most likely require advanced and expensive procedures.
Fix and prevent
Rewriting your flash partition data will fix freezes due to bad blocks. However when freezes start to happen this usually means that some data is already unreadable and therefore - lost. In order to refresh the data the non-destructive write test in badblocks can be used periodically to rewrite the data. This will result in a small speedup of your phone.
Important: You must be careful not to rewrite your partition too often as each flash memory block has a limited number of erase cycles (around 100,000 for SLC and 10,000 for MLC chips). Doing this too often will actually damage your chip in the long term making your phone unusable. I believe that rewriting your /data partition every 6 months without a factory reset should prevent any issues. You do not need to do this if you periodically factory-reset your phone as in such case the data is rewritten anyway.
Step-by-step instructions
DISCLAIMER: This may be a dangerous operation if not executed properly. I do not accept any responsibility for damage that it may cause. Please do this on your own risk.
Prerequisites
Before you begin there are some requirements that need to be fulfilled:
A recent backup of your phone. Remember - there are three kinds of people in this world - those who make backups, those who have not yet lost their data and those morons who still refuse to make backups despite loosing data.
SDS-proof recovery. This is true for all recent recoveries these days, but better make sure. The recovery must also allow ADB connections. While this is true for all recoveries, some have problems with it. In my case I couldn't connect to CWM 6.0.4.4 as ADB kept complaining that the connection is unauthorized. Flashing latest Philz Touch (6.00.8 in my case) solved the problem.
ADB installed on your PC. Please search forum or internet to find instructions.
badblocks binary for Android recovery. Since this utility is not included in the recovery by default you need to obtain it elsewhere. The one I've built is attached to this post.
Instructions
These assume we're working on the /data partition. For other partitions please check their corresponding device node and modify the instructions accordingly.
Reboot to recovery.
Do a nandroid backup if not done before.
Connect to your phone using the USB cable.
Run adb push badblocks sbin/badblocks. In case you have just downloaded the attached ZIP file unzip it first to extract the badblocks binary.
Run adb shell to enter a shell on your phone.
Run chmod 0755 sbin/badblocks to make the badblocks binary executable.
Make sure your /data is not mounted. The quick way is to run mount. It will list all mounted filesystems. If /data on /dev/block/mmcblk0p12 is among them you'll need to unmount it by running: umount /data. If it complains about the filesystem being busy you'll need to reboot the phone and enter recovery again. After that repeat all the steps.
Once you are sure that /data is not mounted run: badblocks -b 4096 -n -t 0xFF -s /dev/block/mmcblk0p12. This command is where the main magic happens. What it does is it reads every sector, fills it with 0xFF bytes and then writes back the original data. This way the data is unchanged, but every block gets rewritten. After running badblocks you should see some percentage indicator about the progress. The operation on /data should take around an hour. Observe any errors that may occur. In my case there were none, but if the flash is heavily worn out there may be some unrecoverable sectors.
In case there are no errors everything should be done. Optionally you may want to trim the /data filesystem as all blocks have been rewritten and the eMMC conroller will think that there are no unused blocks on the device. You can do this by mounting the /data filesystem (mount /data) and running fstrim (fstrim -v /data).
You're good to go. Type exit in the ADB shell and reboot your phone into the system.
In case you do not care for the data on the partition you may run badblocks in destructive mode. In such case in step 7 replace "-n" with "-w" in badblocks command line: badblocks -b 4096 -w -t 0xFF -s /dev/block/mmcblk0p12
The same procedure can be repeated for other filesystems (/system, /cache). I would not recommend doing this on the partitions containing the kernel, recovery, efs or bootloader unless you're asking for trouble.

Reserved for future use.

This should seriously be stickied. Works better than using the dummy file generator that everyone recommends.

Actually, if you're getting SDS freeze on patched firmware with V2 fix (XXEMB2+) it is SDS fix recovering bad block. It's even stated in your kernel message.
This method is good if you want to get rid of misc freezes and fix all blocks at once. Because badblocks forces kernel to rewrite all blocks instead of only one/affected block (during normal freeze).
May be useful, but it's not a magic trick. If somebody gets freezes and kernel can't fix it itself then magic badblocks binary won't help him, unfortunately.

I wonder if you could use e2fsck to mark some permanently broken blocks as bad blocks on the filesystem so that they won't be used for filesystem data. A similar thing used to be done on HDDs. The problem that I see here is that the FTL may relocate that block somewhere and it may end up being reused somewhere else and keep doing damage.
I'm really curious how does the eMMC controller on the I9300 handle permanently broken blocks. Does it have some reservoir of usable blocks (like in modern SSD disks) that can be used as a replacement?

KrissN said:
I wonder if you could use e2fsck to mark some permanently broken blocks as bad blocks on the filesystem so that they won't be used for filesystem data. A similar thing used to be done on HDDs. The problem that I see here is that the FTL may relocate that block somewhere and it may end up being reused somewhere else and keep doing damage.
I'm really curious how does the eMMC controller on the I9300 handle permanently broken blocks. Does it have some reservoir of usable blocks (like in modern SSD disks) that can be used as a replacement?
Click to expand...
Click to collapse
This is actually a very good question, I'm wondering as well. If you're clever enough then you may download my latest ArchiDroid 2.X Experimental ROM with built-in debian and try to discover that with native linux gnu utilities.

JustArchi said:
This is actually a very good question, I'm wondering as well. If you're clever enough then you may download my latest ArchiDroid 2.X Experimental ROM with built-in debian and try to discover that with native linux gnu utilities.
Click to expand...
Click to collapse
Desctructive mode? Will that be equivalent to a lets say factory reset? ie losing all user made data.

Wow, this really helped. I am using SM-G355H (not really made with exynos but with sc8830 from spreadtrum but still by samsung) and was afraid that the phone has failed for good. Though the partition failing was the /system/ not the /data/, OP's badblocks build and instructions definitely fixed it, like magic.
Additional note: To identify which partition is failing
Take note of the failing sector, in my case it's at 1879872
Code:
mmcblk0: error -110 transferring data, sector [B]1879872[/B], nr 256, cmd response 0x900, card status 0xb00
and find the partition using the output of this command where /dev/block/mmcblk0 is the block device
Code:
sgdisk --print [B]/dev/block/mmcblk0[/B]
It will print something like:
Code:
Logical sector size: 512 bytes
Disk identifier (GUID): 52444E41-494F-2044-4D4D-43204449534B
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 7618526
Partitions will be aligned on 512-sector boundaries
Total free space is 21949 sectors (10.7 MiB)
Number Start (sector) End (sector) Size Code Name
1 8192 15871 3.8 MiB 0700 FIXNV1
2 15872 23551 3.8 MiB 0700 FIXNV2
3 23552 27647 2.0 MiB 0700 SBL1
4 27648 31743 2.0 MiB 0700 SBL2
5 31744 41983 5.0 MiB 0700 WDSP
6 41984 52223 5.0 MiB 0700 WDSP_BACKUP
7 52224 72703 10.0 MiB 0700 MODEM
8 72704 93183 10.0 MiB 0700 MODEM_BACKUP
9 93184 93695 256.0 KiB 0700 FOTA_SIG
10 93696 110079 8.0 MiB 0700 OTA
11 110080 117759 3.8 MiB 0700 RUNTIMENV1
12 117760 125439 3.8 MiB 0700 RUNTIMENV2
13 125440 131071 2.8 MiB 0700 PARAM
14 135680 176639 20.0 MiB 0700 efs
15 176640 186879 5.0 MiB 0700 prodnv
16 186880 187391 256.0 KiB 0700 Odin_reserved
17 187392 218111 15.0 MiB 0700 KERNEL
18 218112 248831 15.0 MiB 0700 RECOVERY
19 248832 494591 120.0 MiB 0700 CSC
20 494592 2829311 1.1 GiB 0700 system
21 2829312 2890751 30.0 MiB 0700 HIDDEN
22 2890752 7609343 2.3 GiB 0700 userdata
Find the partition whose start and end sectors contain that failing sector. In my case it's the system since
1879872 in within 494592 to 2829311
Code:
20 [B]494592 2829311[/B] 1.1 GiB 0700 system

[Q] Bricked - and I mean it - VZW VS980... Thoughts?

Well, I was glad to have found the thread that described my case perfectly (OTA brick, no recovery, no download, unable to reset). Thus, I installed Ubuntu, and followed all the steps. When I came to the final step of 'rebooting the phone', my VZW G2 VS980 (I think) was unresponsive to anything. I thought that perhaps it got fully discharged, so I hooked it up to a charger. And - nothing, not even after a few hours of charging. I realize that I used the files that may not be EXACTLY for the version I have, but someone on this thread suggested that these may still work. So - not sure if this was THE death blow...
If before Win8 recognized it as a Qualcomm device (with the attendant smorgasbord of phantom drives), now the error is for a generic 'unrecognized usb device'.
So - it is a complete brick, which does not respond to any hadrware inputs - i.e., matter what buttons are pressed and for how long, there is no response from the device.
Thoughts??
This is a copy from the terminal window, as it stands curently:
/dev/sda /dev/sda2 /dev/sdb1 /dev/sdb5
/dev/sda1 /dev/sdb /dev/sdb2 /dev/sdb6
Disk /dev/sdb: 976773168 sectors, 465.8 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 918B6C9F-D1B0-49D5-8CA2-F14781192942
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 976773134
Partitions will be aligned on 8-sector boundaries
Total free space is 4594 sectors (2.2 MiB)
Number Start (sector) End (sector) Size Code Name
1 63 780764729 372.3 GiB 0700 Microsoft basic data
5 780765184 959999999 85.5 GiB 8300 Linux filesystem
6 960002048 976771071 8.0 GiB 8200 Linux swap