I'm using a SSD for about 460GB of data on an Ubuntu 12.04 server. (The drive holds 560GB, so it is about 88% full.) I have had a program which does a lot of random access to the data (no writing), which I last ran (normally) a few days ago. When I ran it yesterday, it was suddenly significantly slower than before. The program just does random access to the drive.
Previously I could do about 5000 random lookups per second; now I'm only getting about 100. This is slower than doing random access to a HDD. (I benchmarked a few problems last year.)
The only thing I've done recently was to install gcc 4.7 and to upgrade all my packages. But, I tried compiling everything with clang and didn't see a difference.
The drive is formatted as ext4
with the only options being errors=remount-ro
. I tried rebooting the machine and trimming the device but it didn't change things. Profiling the code shows it is spending all its time in open
, close
and mmap
calls. (Note that I'm not calling mmap directly -- I'm using all C
-style fopen
fseek
and fread
calls.)
Any idea what could cause this? I could re-format the drive if there was a chance it would help.
Edit: here is some benchmark data on a 2TB HDD and the 500GB SSD
sudo hdparm -Tt /dev/sdb1
/dev/sdb1:
Timing cached reads: 6814 MB in 2.00 seconds = 3410.05 MB/sec
Timing buffered disk reads: 458 MB in 3.00 seconds = 152.45 MB/sec
sudo hdparm -Tt /dev/sdc1
/dev/sdc1:
Timing cached reads: 6890 MB in 2.00 seconds = 3447.93 MB/sec
Timing buffered disk reads: 780 MB in 3.01 seconds = 259.36 MB/sec
Here is the output from smartctl
:
=== START OF INFORMATION SECTION ===
Model Family: Intel 320 Series SSDs
Device Model: INTEL SSDSA2CW600G3
Serial Number: CVPR140004B7600FGN
LU WWN Device Id: 5 001517 9596df196
Firmware Version: 4PC10362
User Capacity: 600,127,266,816 bytes [600 GB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Sun Jan 26 16:46:53 2014 MST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 1) seconds.
Offline data collection
capabilities: (0x75) SMART execute Offline immediate.
No Auto Offline data collection support.
Abort Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 1) minutes.
Conveyance self-test routine
recommended polling time: ( 1) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 5
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
3 Spin_Up_Time 0x0020 100 100 000 Old_age Offline - 0
4 Start_Stop_Count 0x0030 100 100 000 Old_age Offline - 0
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 17833
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 13
170 Reserve_Block_Count 0x0033 100 100 010 Pre-fail Always - 0
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
183 Runtime_Bad_Block 0x0030 100 100 000 Old_age Offline - 0
184 End-to-End_Error 0x0032 100 100 090 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
192 Unsafe_Shutdown_Count 0x0032 100 100 000 Old_age Always - 5
199 UDMA_CRC_Error_Count 0x0030 100 100 000 Old_age Offline - 0
225 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 3236658
226 Workld_Media_Wear_Indic 0x0032 100 100 000 Old_age Always - 4973
227 Workld_Host_Reads_Perc 0x0032 100 100 000 Old_age Always - 65
228 Workload_Minutes 0x0032 100 100 000 Old_age Always - 1067143
232 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0
233 Media_Wearout_Indicator 0x0032 096 096 000 Old_age Always - 0
241 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 3236658
242 Host_Reads_32MiB 0x0032 100 100 000 Old_age Always - 6437718
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
Note: selective self-test log revision number (0) not 1 implies that no selective self-test has ever been run
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
[Edit: I'm continuing to work on this in the background. When I do resolve it, I will post something here.]
sdc
is the SSD? What SSD model is this? The I/O operations on the HDD didn't slow down? Please also provide the output of ˙sudo smartctl -a /dev/sdc. (If you don't have that program first install with
sudo apt-get install smartmontools`.) This looks like strange situation, hopefully it is not some kind of hardware problem. I would try powering down the machine totally, unplug the SSD from the power, wait a few minutes and then replug it an test again. (if you have physical access to it)sdc
; it is an Intel 320 series. I don't believe the HDD has slowed down -- I can check this in a bit. I've edited my post with more information, although I'm not sure whether the controller is SATA2 or SATA3.