mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-21 15:20:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,054 Commits 32 Branches 296 Tags 152 MiB
1e13b6879c
Commit Graph

873 Commits

Author SHA1 Message Date
Chris Lu
ca998328c2 do not add new volumes when below minFreeSpacePercent 
fix https://github.com/chrislusf/seaweedfs/issues/2017
 2021-04-21 23:11:11 -07:00
Chris Lu
a8114da02d avoid thundering herd effect 
transient errors may cause thundering herd effect to all trying to recover from remove ec shards
 2021-04-21 10:17:12 -07:00
Chris Lu
ae74d8f02a fix error message 
related to https://github.com/chrislusf/seaweedfs/issues/2012
 2021-04-21 01:40:16 -07:00
Chris Lu
283d703d50 adjust text 2021-04-15 11:29:58 -07:00
Chris Lu
8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu
b465095db1 shell: add volume.check.disk to fix inconsistency for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1923
 2021-03-22 00:03:16 -07:00
Chris Lu
102a951377 refactor, split into 2 files 2021-03-21 13:05:53 -07:00
Chris Lu
f6e2566ee5 better logs for cookie mismatch 2021-03-20 12:02:57 -07:00
Chris Lu
35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu
cb423312a4 prevent nil volume message 2021-03-13 11:05:29 -08:00
Chris Lu
972327f966 prevent nil volume nm 2021-03-13 11:04:51 -08:00
Chris Lu
737bde5ab7 refactoring, remove duplicated code 2021-03-09 23:23:01 -08:00
Chris Lu
828f6e9f4d volume: auto add missing vif files 
fix https://github.com/chrislusf/seaweedfs/issues/1878
 2021-03-09 12:09:32 -08:00
qieqieplus
5b16820924 fix: collection stats won't update if all volumes expired at same iteration 2021-03-09 18:49:45 +08:00
Chris Lu
726edab054 avoid nil when closing an index 
fix https://github.com/chrislusf/seaweedfs/issues/1870
 2021-03-07 11:03:09 -08:00
Chris Lu
bdfed16d42 avoid nil exception 
fix https://github.com/chrislusf/seaweedfs/issues/1869
 2021-03-07 01:49:06 -08:00
Chris Lu
e9eddfb8a2 remove wrong writes 2021-03-06 14:26:27 -08:00
Chris Lu
38fc200e56 CRCWriter consistent with CRC 2021-03-06 14:26:27 -08:00
Chris Lu
400de380f4 volume server: support tcp direct put/get/delete 2021-03-05 02:29:38 -08:00
Chris Lu
c46d16b2f3 skip network error when remote volumes start 2021-03-01 01:20:06 -08:00
Chris Lu
6e43e8ce17 add logs for remote file loading 2021-03-01 00:48:59 -08:00
Chris Lu
015d16f43f add vif file versions in case loading superblock fails 2021-03-01 00:48:30 -08:00
Chris Lu
d680676d45 skip already loaded backends 2021-03-01 00:47:03 -08:00
Chris Lu
540441fd38 go fmt 2021-02-28 20:34:14 -08:00
Chris Lu
2cca07b44c cloud tier to non-AWS s3 gateways 2021-02-28 18:59:09 -08:00
Chris Lu
30b30b8fe0 volume.tier.move: passing non-empty disk type 2021-02-22 01:59:03 -08:00
Chris Lu
03c643aa6b fix test 
fileSize need to be divided by 8
 2021-02-20 21:19:21 -08:00
Chris Lu
7635f6b9fa disk file avoid file.Stat() 2021-02-20 20:06:06 -08:00
Chris Lu
a2383b3b12 Revert "Revert "avoid file.Stat()"" 
This reverts commit 7ef4c24f18.
 2021-02-20 12:42:09 -08:00
Chris Lu
7ef4c24f18 Revert "avoid file.Stat()" 
This reverts commit 98c93ca465.
 2021-02-20 12:39:33 -08:00
Chris Lu
1ad3200094 skip seek() when index file writes 2021-02-20 12:39:25 -08:00
Chris Lu
98c93ca465 avoid file.Stat() 
avoid one Syscall, but did not help on performance though
 2021-02-20 09:44:17 -08:00
Chris Lu
a1210d1e8d Revert "volume: avoid file.stat(), file.seek() if possible during writes" 
This reverts commit c78409a598.
 2021-02-19 03:56:27 -08:00
Chris Lu
c78409a598 volume: avoid file.stat(), file.seek() if possible during writes 2021-02-18 23:22:54 -08:00
Chris Lu
73958e357d add descriptive error if no free volumes 2021-02-18 19:10:20 -08:00
bingoohuang
352ac2f271 Merge remote-tracking branch 'origin/master' 2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34 use backticks instead of double quotes to avoid escaped additionally in regex 2021-02-18 14:05:28 +08:00
Chris Lu
3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu
7403cd43c5 only use "" for hdd to avoid two values for the same thing 2021-02-16 10:50:42 -08:00
Chris Lu
ebf320ec22 avoid divided by zero 2021-02-16 05:59:24 -08:00
Chris Lu
3fe628f04e use hdd instead of empty string 2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu
7ce647f27e support customizable disk type 2021-02-13 15:42:42 -08:00
Chris Lu
4bd8a692d8 disk type can be generic tags 2021-02-13 13:50:14 -08:00
Chris Lu
885ca34748 volume: fail fast if idx files are missing 
fix https://github.com/chrislusf/seaweedfs/issues/1796
 2021-02-11 00:44:40 -08:00
Chris Lu
821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
bingoohuang
9e06ef66b9 fix renaming for 5bytes 2021-02-09 09:44:28 +08:00
Chris Lu
a833021132 fix refactoring left over 2021-02-07 23:03:03 -08:00
bingoohuang
7256902fb0 fix typo offset.ToAcutalOffset to offset.ToActualOffset 2021-02-07 12:11:51 +08:00
bingoohuang
94ea3bd3a5 renaming NeedleMapType to NeedleMapKind 2021-02-07 09:00:03 +08:00
Chris Lu
cfb9342a15 avoid concurrent map updates to viper 2021-01-12 02:28:13 -08:00
Chris Lu
039ad101b1 manifest file also need to detect compression 
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753585046
 2021-01-03 01:42:21 -08:00
Chris Lu
4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu
9abf016af1 Merge branch 'master' into support_ssd_volume 2020-12-17 22:30:19 -08:00
Chris Lu
738c4fd203 fix logging when error 
address https://github.com/chrislusf/seaweedfs/pull/1685#issuecomment-747845601
 2020-12-17 19:37:46 -08:00
Chris Lu
daa8157fc2 Merge branch 'master' into support_ssd_volume 2020-12-17 13:05:20 -08:00
Chris Lu
e2076201d7 volume: avoid reprocessing the same volume 
fix https://github.com/chrislusf/seaweedfs/issues/1682
 2020-12-17 13:03:39 -08:00
Chris Lu
986cbdf7d9 Revert "Merge pull request #1683 from qieqieplus/master" 
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
 2020-12-17 12:46:42 -08:00
Chris Lu
f56e6d231e Revert "Merge pull request #1683 from qieqieplus/master" 
This reverts commit 8cb67952db, reversing
changes made to 200e56215a.
 2020-12-17 12:46:20 -08:00
Chris Lu
1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu
1d88865869 passing disk type along 2020-12-16 09:10:14 -08:00
Chris Lu
23014b6810 Merge branch 'master' into support_ssd_volume 2020-12-16 08:26:51 -08:00
chenqieqie
45f902a9b7 smaller critical section 2020-12-16 20:23:51 +08:00
chenqieqie
4e58a4f24e fix race condition when loading volumes concurrently 2020-12-16 18:49:10 +08:00
Chris Lu
2e8dba571b adjust volume server UI 2020-12-14 00:51:57 -08:00
Chris Lu
933f2f4cfd find location with matching disk type 2020-12-13 23:39:00 -08:00
Chris Lu
94525aa0fd allocate volume by disk type 2020-12-13 23:08:21 -08:00
Chris Lu
f6a419c26c disk type configurable for each folder 2020-12-13 22:49:56 -08:00
Chris Lu
0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu
d156c74ec0 volume server set volume type and heartbeat to the master 2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu
ae655033ac adjust logging 2020-12-11 16:57:53 -08:00
Chris Lu
d986c7196d use append time instead of filer's own modification time 
fix https://github.com/chrislusf/seaweedfs/issues/1669
 2020-12-11 16:55:18 -08:00
Chris Lu
d171d9f988 volume: ensure the volume dat and idx files are always closed 
fix https://github.com/chrislusf/seaweedfs/issues/1646
 2020-12-04 21:50:26 -08:00
Chris Lu
cc839f935d better unit detection 2020-12-03 19:23:59 -08:00
Chris Lu
b219ccfe68 Update volume_ttl_test.go 2020-12-03 12:55:44 -08:00
Chris Lu
a83b8e7b01 ttl fix: 3y was converted into 96m 2020-12-03 10:53:29 -08:00
Chris Lu
f3bb645018 file open error 2020-12-01 23:37:49 -08:00
Chris Lu
dc0bc48257 return file open error 2020-12-01 23:36:49 -08:00
Chris Lu
96c48bc8a8 fix test 2020-11-28 03:03:43 -08:00
Chris Lu
2c913dde04 volume: detect and drop volumes with disk IO error 
from Jethro in slack:
is it possible to make the assign request a bit smarter? Currently I’m in the state that a disk failed but all assign request are being send to this volume. It would be cool if the master sees this and stopped using this volume.
e=HTTP(http://x:8089/913,045a782b63176edf) not 200 but 500 Internal Server Error
Body={"size":740167,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"ee4381e202212ff3aee647704c036689"}
e=HTTP(http://x:8089/913,045a782c90240077) not 200 but 500 Internal Server Error
Body={"size":792779,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"c43463ccc11eb6eb2fc306f407a6a953"}
e=HTTP(http://x:8089/913,045a782e6b7901ea) not 200 but 500 Internal Server Error
Body={"size":3962392,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"04c91198e9b276c81f11dbf189af5d28"}
 2020-11-28 00:09:29 -08:00
Chris Lu
9ac4935f22 read from volume index file directly instead of open a separate file 
fix https://github.com/chrislusf/seaweedfs/issues/1640

read from volume index file directly instead of open a separate file,
to ensure reading latest index entries.
 2020-11-27 16:18:48 -08:00
Chris Lu
6d30b21b10 volume: add "-dir.idx" option for separate index storage 
fix https://github.com/chrislusf/seaweedfs/issues/1265
 2020-11-27 03:17:10 -08:00
Chris Lu
3c229eb677 refactor 2020-11-26 17:21:55 -08:00
Chris Lu
c0fb4a3766 add comments 2020-11-26 15:22:42 -08:00
Chris Lu
6783ae5f8f rename 2020-11-26 15:19:43 -08:00
Chris Lu
97f706f35b just refactoring 2020-11-26 15:08:05 -08:00
Chris Lu
b11449f955 pass in dir.idx parameter to DiskLocation 2020-11-26 14:59:03 -08:00
Chris Lu
0da7ecfd29 go fmt 2020-11-26 11:22:30 -08:00
Chris Lu
92f906b6fc remove zstd 
fix https://github.com/chrislusf/seaweedfs/issues/1629
 2020-11-21 13:06:45 -08:00
Chris Lu
1dd3a6ac36 avoid verbose logs 2020-11-09 15:56:11 -08:00
Chris Lu
0dafcf1f5a volume: detect max volume count changes based on disk usage 
fix https://github.com/chrislusf/seaweedfs/issues/1594
 2020-11-03 14:43:17 -08:00
Chris Lu
de86945aeb go fmt 2020-10-31 16:45:38 -07:00
Chris Lu
9708df47d1 ensure offset in the index file is positive 2020-10-31 15:50:01 -07:00
Konstantin Lebedev
46303c36bf When the volume server is stopped, the master server immediately sees the deletion of volumes 2020-10-31 02:48:25 +05:00
Chris Lu
6560ac6466 volume loading: trim out unreachable idx file content 
fix https://github.com/chrislusf/seaweedfs/issues/1583
 2020-10-29 22:25:23 -07:00
Konstantin Lebedev
1ad1b8c4f6 collection Volume ReadOnly Count with detailed status 2020-10-28 17:38:26 +05:00
Chris Lu
744c5594da volume: automatically trim out unreachable entries 2020-10-28 01:14:39 -07:00
Chris Lu
3e925faddd trim out name extension 2020-10-28 01:14:05 -07:00
shibinbin
a67eb1afa1 fix: restart volumeserver sometime occurred out of memory error 2020-10-28 12:03:06 +08:00
Chris Lu
53c3aad875 volume: add a note file to avoid incomplete volume files 
fix https://github.com/chrislusf/seaweedfs/issues/1567
 2020-10-27 15:56:49 -07:00
Chris Lu
06c15ab35c volume: add special handling for .dat larger than 32GB 2020-10-27 13:11:56 -07:00
Konstantin Lebedev
2fb1fce8a0 avoid old values when missing read only flags 2020-10-25 16:21:40 +05:00
Chris Lu
9104cfa744 reduce locks 2020-10-24 19:40:35 -07:00
Chris Lu
e9d40b80b2 less verbose logs 2020-10-24 09:42:54 -07:00
Chris Lu
575d7952a1 add available resource stats 
fix https://github.com/chrislusf/seaweedfs/issues/1555
 2020-10-22 09:13:47 -07:00
Chris Lu
4ff2ceee33 UI fix on rendering EC volumes 
addressing UI problem with https://github.com/chrislusf/seaweedfs/issues/1551
 2020-10-21 22:05:58 -07:00
Konstantin Lebedev
dc2e13092d add number of read only volumes metric 2020-10-15 15:48:40 +05:00
Chris Lu
bd8b9b0c2e some changes to prepare fix after failed reads 2020-10-08 18:33:06 -07:00
Chris Lu
d522df0ecd adjust log level 2020-10-04 01:35:32 -07:00
Chris Lu
a1c01d716b volume: avoid deadlock when deleting volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1501
 2020-10-01 07:10:03 -07:00
Chris Lu
103fafe00b weed export: print out [start,stop) content range 2020-09-14 22:57:23 -07:00
Chris Lu
f2723c1bc8 do not idx file format 
revert c9ab8d05fa
 2020-09-12 12:42:36 -07:00
Chris Lu
446e476a11 go fmt 2020-09-12 04:08:03 -07:00
Chris Lu
ea26a98753 volume: validate volume correctness if last entry is a deletion 2020-09-12 04:07:04 -07:00
Chris Lu
9d4bdfcfdf fix volume integrity checking 2020-09-11 11:34:10 -07:00
Chris Lu
401ccf1509 fix test 2020-09-10 15:13:23 -07:00
Chris Lu
c9ab8d05fa fixes for reading deleted fid 2020-09-10 14:42:52 -07:00
James Hartig
8e54e34576 volume: Don't unmount before deleting volume in copy 
If we unmount first and then delete, the delete fails because the volume
was unmounted. Delete ends up doing the same thing as the unmount anyways.
 2020-09-01 22:00:07 -04:00
Chris Lu
38e06d783d volume: check disk space before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1440
 2020-08-31 18:10:53 -07:00
Chris Lu
dbb10e0f0b testing with more than 30days 2020-08-31 17:59:29 -07:00
Chris Lu
99ecf63276 go fmt 2020-08-29 22:28:33 -07:00
Chris Lu
ab759f0ec2 erasure coding: fix EC error if multiple disks are configured in one volume server 2020-08-26 09:16:58 -07:00
Chris Lu
d1cf39f180 fix logging 2020-08-26 08:55:15 -07:00
James Hartig
3ccfa4c6ad Added VolumeMarkWritable and VolumeStatus grpc methods 
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
 2020-08-19 11:42:56 -04:00
Chris Lu
3b4b1d4a77 fix tests 2020-08-19 01:37:56 -07:00
Chris Lu
ed4b43b419 adjust logs 2020-08-18 23:42:09 -07:00
Chris Lu
fe01191b5b support read option readDeleted=true 2020-08-18 19:22:16 -07:00
Chris Lu
6ccd7f0a4d refactoring 2020-08-18 18:01:37 -07:00
Chris Lu
c026eb0592 refactoring 2020-08-18 17:39:29 -07:00
Chris Lu
7e91ae592c pass in option to read deleted entries 
not working yet
 2020-08-18 17:37:26 -07:00
Chris Lu
ee11d98650 refactoring 2020-08-18 17:35:19 -07:00
Chris Lu
6a92f0bc7a refactoring to typed Size 
Go is amazing with refactoring!
 2020-08-18 17:04:28 -07:00
Chris Lu
627b081b67 adjust logs 2020-08-16 16:32:22 -07:00
Chris Lu
c03bb180eb fix error reporting 2020-08-14 00:44:02 -07:00
Chris Lu
ab6e5c0dc4 adjust error message 2020-08-08 10:18:32 -07:00
Chris Lu
20e2ac1add filer: store md5 metadata for files uploaded by filer 
fix https://github.com/chrislusf/seaweedfs/issues/1412
 2020-08-06 10:04:17 -07:00
Chris Lu
4ecfa9879d volume: report Content-MD5 in response header 2020-08-06 05:22:53 -07:00
Chris Lu
cbd80253e3 better needle id format 2020-08-04 22:09:07 -07:00
Chris Lu
f89186a3cf better logging 2020-08-01 11:19:57 -07:00
Chris Lu
943fa4ea0c fix compilation error 2020-07-22 16:11:17 -07:00
Chris Lu
49446af516 simplify 2020-07-22 13:43:54 -07:00
James Hartig
0be3bb4a87 idx: Switch WalkIndexFile to accept generic io.ReaderAt 
This is necessary to use it with in-memory idx files.
 2020-07-22 15:01:16 -04:00
Chris Lu
97d97f3528 go code can read and write chunk manifest 2020-07-19 17:59:43 -07:00
Chris Lu
f43146b237 resolve directories if containing home directory 2020-07-16 22:50:14 -07:00
Chris Lu
ddec7b2bb9 go fmt 2020-07-14 07:34:16 -07:00
Chris Lu
f2e8ad6241 refactoring 2020-07-11 06:35:54 -07:00
李远军
6608cb5f43 Revert "remove fixJpgOrientation" 
This reverts commit de5ca9b2
 2020-07-10 10:08:36 +08:00
Chris Lu
4bed1663e2 adjust log level 2020-07-07 23:17:31 -07:00
Chris Lu
4d1484628a refactoring 2020-07-03 16:41:30 -07:00
Chris Lu
faa5c2e89a refactoring 2020-07-03 16:34:31 -07:00
Chris Lu
20d9cfae86 volume: following md5 base64 encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1374
 2020-06-28 07:25:50 -07:00
Chris Lu
212c64241b checking ecx file 2020-06-25 16:36:06 -07:00
Chris Lu
9fd7cdadf1 fix 2020-06-25 10:45:34 -07:00
Chris Lu
3b638d3994 add more ec encoding logging 2020-06-25 09:43:38 -07:00
Chris Lu
32a1f22730 more logs 2020-06-24 22:07:53 -07:00
Chris Lu
2ff37ccdbd testing parsing the upload 
The mime type is always the value passed in.

Compress or not depends on the content detection, file name extension, and compression ratio.

If the content is already compressed, need to know the content size.
 2020-06-24 11:39:12 -07:00
Chris Lu
c21f4ebfee
Merge pull request #1371 from Kimbsen/content_md5_validation 
Optional md5 validation of uploads
 2020-06-24 11:33:44 -07:00
Kimbsen
4eeab2a379 remove unused imports 2020-06-24 13:37:00 +02:00
Kimbsen
ffddecebef do md5 validation AFTER decompression 2020-06-24 13:35:13 +02:00
Chris Lu
fe60db404a prepare for zstd 2020-06-23 09:12:02 -07:00
Kimbsen
6b1e93ba0b check request header not part header for content-encoding 2020-06-23 17:04:24 +02:00
Kimbsen
38626cb584 don't forget the last few bytes 2020-06-23 16:53:49 +02:00
Kimbsen
8d1f6cd567 added optional md5 verification of uploaded data 2020-06-23 16:24:11 +02:00
Chris Lu
de5ca9b258 remove fixJpgOrientation 2020-06-20 12:50:40 -07:00
Chris Lu
3080c197e3 rename UnCompressData to DecompressData 2020-06-20 08:16:16 -07:00
Chris Lu
e912fd15e3 renaming 2020-06-19 22:45:27 -07:00
Chris Lu
16fe132a20 detect mime type before replicating to other volume servers 2020-06-19 22:11:36 -07:00
Chris Lu
ae1994cbc1 erasure coding: fix cases where there are no .ecj files 2020-06-18 09:52:35 -07:00
Chris Lu
7ccc31853c hard limit volume file size 2020-06-16 00:27:48 -07:00
Chris Lu
bdbe5ddbe9 volume server: add a fix for dead lock 
https://github.com/chrislusf/seaweedfs/issues/1353
 2020-06-08 08:12:59 -07:00
Chris Lu
45a9d852d3 go fmt 2020-06-05 15:27:10 -07:00
Evgenii Kozlov
2410ffbd79 Name fix 2020-06-05 19:00:38 +03:00
Evgenii Kozlov
0e0db70f55 Set volumes ReadOnly if low free disk space 2020-06-05 18:18:15 +03:00
Chris Lu
ef2b3a0801 leveldb recover error handling 2020-05-26 00:03:44 -07:00
Chris Lu
e5a0787653 filer: leveldb store add auto recovery, report correct error 
fix https://github.com/chrislusf/seaweedfs/issues/1326
 2020-05-22 10:54:42 -07:00
Chris Lu
7ff2b0121d specify constant data type to uint64 instead of default int 
fix https://github.com/chrislusf/seaweedfs/issues/1324
 2020-05-19 07:55:23 -07:00
Chris Lu
11b33c0760 fix compilation problem 2020-05-18 10:53:31 -07:00
zhangsong
b161baa3f0 fix volume vacuum test unit 2020-05-07 19:58:06 +08:00
zhangsong
3703c7e176 write request keep compatible with before 2020-05-06 21:35:32 +08:00
zhangsong
f9e8702bb4 use async write to persistent file to disk - part1 2020-05-04 17:39:44 +08:00
Chris Lu
6190fd665d printout error 2020-04-27 12:41:31 -07:00
Chris Lu
bafa95045b volume: deletion checks all disk locations 
fix https://github.com/chrislusf/seaweedfs/issues/1283
 2020-04-21 14:49:58 -07:00
Chris Lu
7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu
e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
Chris Lu
c8ca234773 refactoring 2020-04-11 14:27:25 -07:00
Chris Lu
df9d538044 rename function 2020-04-11 14:19:44 -07:00
Chris Lu
9fa065f600 typo in logs 2020-04-09 23:43:09 -07:00
Chris Lu
e63a79ade8 better handle lock in case of exception 2020-03-24 18:41:25 -07:00
Chris Lu
3137777d83 volume: automatically detect max volume count 2020-03-22 16:21:42 -07:00
Chris Lu
cbfe31a9a8 idx file sync before compaction 2020-03-20 23:38:46 -07:00
Chris Lu
81797a059a volume: sync volume file right before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1237
 2020-03-19 23:54:52 -07:00
Chris Lu
709f231e23 tiered storage: add s3 endpoint for private s3 implementation 
fix https://github.com/chrislusf/seaweedfs/issues/1238
 2020-03-19 21:13:56 -07:00
Chris Lu
b964bbab3d fix compilation 2020-03-17 10:01:24 -07:00
Chris Lu
c3cb6fa1d7 volume: compaction can cause readonly volumes 
address https://github.com/chrislusf/seaweedfs/issues/1233
 2020-03-17 09:43:57 -07:00
Chris Lu
22400c6633 consistent gzip logic 
local store gzip same as replicated writes
 2020-03-15 04:15:40 -07:00
Chris Lu
560df51def refactoring 2020-03-15 03:11:26 -07:00
Chris Lu
d439d83772 volume: follow compactionBytePerSecond 
related to https://github.com/chrislusf/seaweedfs/issues/1108
 2020-03-11 10:32:17 -07:00
Chris Lu
d3e1ef64b3 ensure error checking on windows 2020-03-10 13:33:26 -07:00
Chris Lu
88a110e67e release file handle 2020-03-10 13:32:22 -07:00
Chris Lu
43e6261579 volume: fix memory leak 2020-03-10 13:31:23 -07:00
Chris Lu
0871d2cff0 volume: fix memory leak during compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1222
 2020-03-09 22:29:02 -07:00
Chris Lu
89eb05b50f filer: support TTL for all filer stores 2020-03-09 01:02:01 -07:00
Chris Lu
9b3109a5d8 filer: processing all response headers, no pass through to volume server 
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
 2020-03-08 15:42:44 -07:00
Chris Lu
1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00
Chris Lu
ea1169dc80 filer cipher: single chunk http POST and PUT and read 2020-03-07 06:08:08 -08:00
Chris Lu
7335e62199 volume: PUT also conditionally gzip compress 2020-03-01 23:37:02 -08:00
Chris Lu
151114ff59 volume: fix readonly status reporting 2020-03-01 17:57:37 -08:00
Chris Lu
f87f2045b3 delete old file first on windows 
fix https://github.com/chrislusf/seaweedfs/issues/1210
 2020-02-27 00:59:35 -08:00
Chris Lu
757c7d67ad avoid nil DataBackend 2020-02-27 00:07:25 -08:00
Chris Lu
430b5a49e2 unused code 2020-02-26 19:35:00 -08:00
Chris Lu
ed0acd1722 go fmt 2020-02-26 16:52:57 -08:00
Chris Lu
97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
panyc16
7b3764fd9e make loadExistingVolume() return bool 2020-02-13 15:19:00 +08:00
Chris Lu
8d94564f41 refactor 2020-02-04 21:16:34 -08:00
Chris Lu
c1288e9eb4 volume: sdx generation uses memdb instead of compactMap 
fix https://github.com/chrislusf/seaweedfs/issues/1194
 2020-02-04 21:12:09 -08:00
Chris Lu
08e4702542 sync before closing index file 
merge from 0181f87d91
 2020-02-04 10:38:12 -08:00
Chris Lu
3a35632d58 sync before closing file 
merge changes from 1747fc2d52
 2020-02-04 10:37:14 -08:00
Chris Lu
40ae533fa3 shell: add volume.configure.replication to change replication for a volume 
fix https://github.com/chrislusf/seaweedfs/issues/1192
 2020-02-02 15:37:23 -08:00
Chris Lu
d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu
72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu
b6e6ca8595 adjust logs 2020-01-24 20:06:58 -08:00
Chris Lu
c8b2dac6c1 volume: avoid sharing volume dat file handle 
possibly help on https://github.com/chrislusf/seaweedfs/issues/1184
 2020-01-21 21:18:01 -08:00
Chris Lu
f8a20ef35e add the old way to compact as a comment 2020-01-08 09:45:42 -08:00
Chris Lu
288baf37fd saving .vif files correctly 2020-01-08 09:45:26 -08:00
Chris Lu
acf7ca7b93 volume: fix compaction 2020-01-08 09:45:03 -08:00
Chris Lu
943f4986ef fix possible nil 2020-01-08 08:49:18 -08:00
Chris Lu
9995d3bcb5 remove println 2020-01-07 21:49:28 -08:00
Chris Lu
3eafec4b29 volume: add option to limit file size 2020-01-03 00:37:26 -08:00
Chris Lu
672868b460 always create .vif file 2019-12-28 21:52:06 -08:00
Chris Lu
b6f7033478 go fmt 2019-12-28 21:37:29 -08:00
Chris Lu
9134092a8d endure creating .vif files 2019-12-28 21:36:15 -08:00
Chris Lu
0cc8347a3a go fmt 2019-12-28 21:13:10 -08:00
Chris Lu
37b64a50b4 ec: generate and copy .vif file 2019-12-28 12:44:59 -08:00
Chris Lu
c06f7eb48a load volume info from .vif file, use superblock as a backup 2019-12-28 12:28:58 -08:00
Chris Lu
f4a74e03d1 refactoring: separating .vif from tier file loading 2019-12-28 11:35:27 -08:00
Chris Lu
2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu
e8b357fd53 rename from .tier to .vif 2019-12-28 11:17:39 -08:00
Chris Lu
48d28d3eb2 tier: support remote file attributes and remember the file extension 2019-12-25 21:37:24 -08:00
Chris Lu
1346437d71
Merge pull request #1171 from stlpmo-jn/redundant_type_conversion 
remove the redundant type conversion
 2019-12-25 17:02:05 -08:00
Chris Lu
35393b4a02 cleanly destroy remote files 2019-12-25 16:17:58 -08:00
Chris Lu
eecad6062a go fmt 2019-12-25 09:56:42 -08:00
Chris Lu
d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
stlpmo
f3de4b6c18 remove the redundant type conversion 2019-12-25 10:13:45 +08:00
Chris Lu
9ff72f616a go fmt 2019-12-24 14:56:16 -08:00
Chris Lu
efd2f50ede compaction changed to .idx based deletion 2019-12-24 14:55:50 -08:00
Chris Lu
d8b39fe92a testing 2019-12-24 11:29:26 -08:00
Chris Lu
f42b5bd0f5 rename .sdb to .sdx 2019-12-24 10:19:12 -08:00
Chris Lu
abffe857a1 change btree map to in memory level db 2019-12-24 10:18:56 -08:00
Chris Lu
72a561ab7c refactoring 2019-12-24 08:13:01 -08:00
Chris Lu
409a3fe41f
Merge pull request #1168 from stlpmo-jn/optimize_DeleteCollectionFromDiskLocation 
decouple the volume.Destroy() from the operation of unmountVolume()
 2019-12-24 07:08:04 -08:00
stlpmo
38e4b79125 decouple the volume.Destroy() from the operation of unmountVolume() 2019-12-24 17:20:34 +08:00
stlpmo
f300c35266 resolve issue : when the host disk full, the volume server will core dump 2019-12-24 16:36:15 +08:00
Chris Lu
a18f62bbe7 only copy required shards 2019-12-23 18:06:13 -08:00
Chris Lu
8a2d529759 go fmt 2019-12-23 12:48:53 -08:00
Chris Lu
09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu
ab966d7192 refactoring variable name 2019-12-19 00:44:46 -08:00
Chris Lu
f61de28c69 volume: add deletion capability for previously readonly volumes 2019-12-19 00:42:46 -08:00
Chris Lu
deb03ecfa3 remove duplicated checking 
already checked in store.go
 2019-12-18 23:59:53 -08:00
Chris Lu
ef3ae3cd41 clean up sorted index file 2019-12-18 20:59:15 -08:00
Chris Lu
39810a9951 tiered volume is read only 2019-12-18 01:26:22 -08:00
Chris Lu
58f88e530c volume: use sorted index map for readonly volumes 2019-12-18 01:21:21 -08:00
Chris Lu
356bd1b629 adjust text 2019-12-10 22:09:29 -08:00
Chris Lu
10bd3c6b4b refactoring 2019-12-08 19:44:16 -08:00
Chris Lu
2b8e20f122 remove unused functions 2019-12-08 18:33:25 -08:00
Chris Lu
f38f90b7ea cleanly remove .tier volume file 2019-12-06 22:19:47 -08:00
Chris Lu
6827cabfbb volume: dataFileAccessLock change to RW Lock 2019-12-06 06:59:57 -08:00
Chris Lu
e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
Chris Lu
7ae8b1cc86 show volume info in volume.list in weed shell 2019-12-02 23:38:56 -08:00
Chris Lu
1becbce657 display remote volumes on volume server ui page 2019-12-02 23:23:54 -08:00
Chris Lu
70648d35ad go fmt 2019-12-02 20:49:58 -08:00
Chris Lu
6383b45bd0 add lock variable 2019-12-02 20:49:50 -08:00
Chris Lu
caae543a9f fix test 2019-12-02 15:54:24 -08:00
Chris Lu
ec8de250e2 tiered storage: can copy to s3, read from s3 
master not aware tiered volume yet, file assigning is not working yet
 2019-12-02 15:08:31 -08:00
Chris Lu
0da7b894cc pass backend config from master to volume servers 2019-11-29 01:05:09 -08:00
Chris Lu
61bc1d6ffd tweaking 2019-11-28 18:47:51 -08:00
Chris Lu
f60154f330 master load backend storage config from master.toml 2019-11-28 18:33:18 -08:00
Chris Lu
ee2072dff9 volume: avoid inifinite loop reading ec volume info if error happens 2019-11-27 23:07:53 -08:00
Chris Lu
0f9ba84274 s3 2019-11-27 03:09:45 -08:00
Chris Lu
e4c63ff2f8 go fmt 2019-11-22 14:51:09 -08:00
Chris Lu
9711a6ffaa WIP 2019-11-18 19:24:37 -08:00
chenwanli
295f3beed5 rm ldb and bdb directory 2019-11-19 09:35:06 +08:00
Chris Lu
c6e8225a20 adjust memory mapped file size 
related to https://github.com/chrislusf/seaweedfs/pull/1087
 2019-11-12 21:05:48 -08:00
Lei Liu
46755ea1e1 fix master maintenance logic 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-11-12 14:46:10 +08:00
Chris Lu
85f8649320 refactor memory mapped file into backend storage 2019-11-09 00:10:59 -08:00
Chris Lu
c5c1d83d91 Merge branch 'master' into refactoring_dat_backend 2019-11-08 22:54:41 -08:00
Chris Lu
c34ffed43f go fmt 2019-11-08 22:47:50 -08:00
Chris Lu
84c503c6a7 adjust ec reading log level 2019-11-08 22:41:02 -08:00
zhangsong
1dd101f782 use read lock to avoid io hang during heartbeat 2019-11-08 20:27:55 +08:00
Chris Lu
db30a46050 skip memory mapped file when compacting 2019-10-29 23:34:38 -07:00
Chris Lu
5b950c735e Merge branch 'master' into refactoring_dat_backend 2019-10-29 23:18:41 -07:00
Chris Lu
57e441d67b fix compaction logic 2019-10-29 23:18:01 -07:00
Chris Lu
b7156291a8 adjust variable names 2019-10-29 23:16:43 -07:00
Chris Lu
4b5ba4927f adjust NewDiskFile() paramaters 2019-10-29 22:37:36 -07:00
Chris Lu
cd7ce720c7 fix test 2019-10-29 00:37:15 -07:00
Chris Lu
19b6a16003 changed from os.file to backend.DataStorageBackend 2019-10-29 00:35:16 -07:00
Chris Lu
7a51a9a582 refactor: simplifying to ReadAt() and WriteAt() 2019-10-25 00:11:25 -07:00
Chris Lu
46ed2ca902 refactoring 2019-10-24 23:41:32 -07:00
Chris Lu
f937933b3d refactoring 2019-10-23 22:25:53 -07:00