mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-02 16:39:16 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,871 Commits 32 Branches 296 Tags 149 MiB
c7e7431fd3
Commit Graph

559 Commits

Author SHA1 Message Date
Chris Lu
ea26a98753 volume: validate volume correctness if last entry is a deletion 2020-09-12 04:07:04 -07:00
Chris Lu
9d4bdfcfdf fix volume integrity checking 2020-09-11 11:34:10 -07:00
Chris Lu
401ccf1509 fix test 2020-09-10 15:13:23 -07:00
Chris Lu
c9ab8d05fa fixes for reading deleted fid 2020-09-10 14:42:52 -07:00
James Hartig
8e54e34576 volume: Don't unmount before deleting volume in copy 
If we unmount first and then delete, the delete fails because the volume
was unmounted. Delete ends up doing the same thing as the unmount anyways.
 2020-09-01 22:00:07 -04:00
Chris Lu
38e06d783d volume: check disk space before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1440
 2020-08-31 18:10:53 -07:00
Chris Lu
dbb10e0f0b testing with more than 30days 2020-08-31 17:59:29 -07:00
Chris Lu
99ecf63276 go fmt 2020-08-29 22:28:33 -07:00
Chris Lu
ab759f0ec2 erasure coding: fix EC error if multiple disks are configured in one volume server 2020-08-26 09:16:58 -07:00
Chris Lu
d1cf39f180 fix logging 2020-08-26 08:55:15 -07:00
James Hartig
3ccfa4c6ad Added VolumeMarkWritable and VolumeStatus grpc methods 
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
 2020-08-19 11:42:56 -04:00
Chris Lu
3b4b1d4a77 fix tests 2020-08-19 01:37:56 -07:00
Chris Lu
ed4b43b419 adjust logs 2020-08-18 23:42:09 -07:00
Chris Lu
fe01191b5b support read option readDeleted=true 2020-08-18 19:22:16 -07:00
Chris Lu
6ccd7f0a4d refactoring 2020-08-18 18:01:37 -07:00
Chris Lu
c026eb0592 refactoring 2020-08-18 17:39:29 -07:00
Chris Lu
7e91ae592c pass in option to read deleted entries 
not working yet
 2020-08-18 17:37:26 -07:00
Chris Lu
ee11d98650 refactoring 2020-08-18 17:35:19 -07:00
Chris Lu
6a92f0bc7a refactoring to typed Size 
Go is amazing with refactoring!
 2020-08-18 17:04:28 -07:00
Chris Lu
627b081b67 adjust logs 2020-08-16 16:32:22 -07:00
Chris Lu
c03bb180eb fix error reporting 2020-08-14 00:44:02 -07:00
Chris Lu
ab6e5c0dc4 adjust error message 2020-08-08 10:18:32 -07:00
Chris Lu
20e2ac1add filer: store md5 metadata for files uploaded by filer 
fix https://github.com/chrislusf/seaweedfs/issues/1412
 2020-08-06 10:04:17 -07:00
Chris Lu
4ecfa9879d volume: report Content-MD5 in response header 2020-08-06 05:22:53 -07:00
Chris Lu
cbd80253e3 better needle id format 2020-08-04 22:09:07 -07:00
Chris Lu
f89186a3cf better logging 2020-08-01 11:19:57 -07:00
Chris Lu
943fa4ea0c fix compilation error 2020-07-22 16:11:17 -07:00
Chris Lu
49446af516 simplify 2020-07-22 13:43:54 -07:00
James Hartig
0be3bb4a87 idx: Switch WalkIndexFile to accept generic io.ReaderAt 
This is necessary to use it with in-memory idx files.
 2020-07-22 15:01:16 -04:00
Chris Lu
97d97f3528 go code can read and write chunk manifest 2020-07-19 17:59:43 -07:00
Chris Lu
f43146b237 resolve directories if containing home directory 2020-07-16 22:50:14 -07:00
Chris Lu
ddec7b2bb9 go fmt 2020-07-14 07:34:16 -07:00
Chris Lu
f2e8ad6241 refactoring 2020-07-11 06:35:54 -07:00
李远军
6608cb5f43 Revert "remove fixJpgOrientation" 
This reverts commit de5ca9b2
 2020-07-10 10:08:36 +08:00
Chris Lu
4bed1663e2 adjust log level 2020-07-07 23:17:31 -07:00
Chris Lu
4d1484628a refactoring 2020-07-03 16:41:30 -07:00
Chris Lu
faa5c2e89a refactoring 2020-07-03 16:34:31 -07:00
Chris Lu
20d9cfae86 volume: following md5 base64 encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1374
 2020-06-28 07:25:50 -07:00
Chris Lu
212c64241b checking ecx file 2020-06-25 16:36:06 -07:00
Chris Lu
9fd7cdadf1 fix 2020-06-25 10:45:34 -07:00
Chris Lu
3b638d3994 add more ec encoding logging 2020-06-25 09:43:38 -07:00
Chris Lu
32a1f22730 more logs 2020-06-24 22:07:53 -07:00
Chris Lu
2ff37ccdbd testing parsing the upload 
The mime type is always the value passed in.

Compress or not depends on the content detection, file name extension, and compression ratio.

If the content is already compressed, need to know the content size.
 2020-06-24 11:39:12 -07:00
Chris Lu
c21f4ebfee
Merge pull request #1371 from Kimbsen/content_md5_validation 
Optional md5 validation of uploads
 2020-06-24 11:33:44 -07:00
Kimbsen
4eeab2a379 remove unused imports 2020-06-24 13:37:00 +02:00
Kimbsen
ffddecebef do md5 validation AFTER decompression 2020-06-24 13:35:13 +02:00
Chris Lu
fe60db404a prepare for zstd 2020-06-23 09:12:02 -07:00
Kimbsen
6b1e93ba0b check request header not part header for content-encoding 2020-06-23 17:04:24 +02:00
Kimbsen
38626cb584 don't forget the last few bytes 2020-06-23 16:53:49 +02:00
Kimbsen
8d1f6cd567 added optional md5 verification of uploaded data 2020-06-23 16:24:11 +02:00
Chris Lu
de5ca9b258 remove fixJpgOrientation 2020-06-20 12:50:40 -07:00
Chris Lu
3080c197e3 rename UnCompressData to DecompressData 2020-06-20 08:16:16 -07:00
Chris Lu
e912fd15e3 renaming 2020-06-19 22:45:27 -07:00
Chris Lu
16fe132a20 detect mime type before replicating to other volume servers 2020-06-19 22:11:36 -07:00
Chris Lu
ae1994cbc1 erasure coding: fix cases where there are no .ecj files 2020-06-18 09:52:35 -07:00
Chris Lu
7ccc31853c hard limit volume file size 2020-06-16 00:27:48 -07:00
Chris Lu
bdbe5ddbe9 volume server: add a fix for dead lock 
https://github.com/chrislusf/seaweedfs/issues/1353
 2020-06-08 08:12:59 -07:00
Chris Lu
45a9d852d3 go fmt 2020-06-05 15:27:10 -07:00
Evgenii Kozlov
2410ffbd79 Name fix 2020-06-05 19:00:38 +03:00
Evgenii Kozlov
0e0db70f55 Set volumes ReadOnly if low free disk space 2020-06-05 18:18:15 +03:00
Chris Lu
ef2b3a0801 leveldb recover error handling 2020-05-26 00:03:44 -07:00
Chris Lu
e5a0787653 filer: leveldb store add auto recovery, report correct error 
fix https://github.com/chrislusf/seaweedfs/issues/1326
 2020-05-22 10:54:42 -07:00
Chris Lu
7ff2b0121d specify constant data type to uint64 instead of default int 
fix https://github.com/chrislusf/seaweedfs/issues/1324
 2020-05-19 07:55:23 -07:00
Chris Lu
11b33c0760 fix compilation problem 2020-05-18 10:53:31 -07:00
zhangsong
b161baa3f0 fix volume vacuum test unit 2020-05-07 19:58:06 +08:00
zhangsong
3703c7e176 write request keep compatible with before 2020-05-06 21:35:32 +08:00
zhangsong
f9e8702bb4 use async write to persistent file to disk - part1 2020-05-04 17:39:44 +08:00
Chris Lu
6190fd665d printout error 2020-04-27 12:41:31 -07:00
Chris Lu
bafa95045b volume: deletion checks all disk locations 
fix https://github.com/chrislusf/seaweedfs/issues/1283
 2020-04-21 14:49:58 -07:00
Chris Lu
7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu
e4af63a721 volume server: accept fsync=true in write requests 2020-04-11 21:39:16 -07:00
Chris Lu
c8ca234773 refactoring 2020-04-11 14:27:25 -07:00
Chris Lu
df9d538044 rename function 2020-04-11 14:19:44 -07:00
Chris Lu
9fa065f600 typo in logs 2020-04-09 23:43:09 -07:00
Chris Lu
e63a79ade8 better handle lock in case of exception 2020-03-24 18:41:25 -07:00
Chris Lu
3137777d83 volume: automatically detect max volume count 2020-03-22 16:21:42 -07:00
Chris Lu
cbfe31a9a8 idx file sync before compaction 2020-03-20 23:38:46 -07:00
Chris Lu
81797a059a volume: sync volume file right before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1237
 2020-03-19 23:54:52 -07:00
Chris Lu
709f231e23 tiered storage: add s3 endpoint for private s3 implementation 
fix https://github.com/chrislusf/seaweedfs/issues/1238
 2020-03-19 21:13:56 -07:00
Chris Lu
b964bbab3d fix compilation 2020-03-17 10:01:24 -07:00
Chris Lu
c3cb6fa1d7 volume: compaction can cause readonly volumes 
address https://github.com/chrislusf/seaweedfs/issues/1233
 2020-03-17 09:43:57 -07:00
Chris Lu
22400c6633 consistent gzip logic 
local store gzip same as replicated writes
 2020-03-15 04:15:40 -07:00
Chris Lu
560df51def refactoring 2020-03-15 03:11:26 -07:00
Chris Lu
d439d83772 volume: follow compactionBytePerSecond 
related to https://github.com/chrislusf/seaweedfs/issues/1108
 2020-03-11 10:32:17 -07:00
Chris Lu
d3e1ef64b3 ensure error checking on windows 2020-03-10 13:33:26 -07:00
Chris Lu
88a110e67e release file handle 2020-03-10 13:32:22 -07:00
Chris Lu
43e6261579 volume: fix memory leak 2020-03-10 13:31:23 -07:00
Chris Lu
0871d2cff0 volume: fix memory leak during compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1222
 2020-03-09 22:29:02 -07:00
Chris Lu
89eb05b50f filer: support TTL for all filer stores 2020-03-09 01:02:01 -07:00
Chris Lu
9b3109a5d8 filer: processing all response headers, no pass through to volume server 
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
 2020-03-08 15:42:44 -07:00
Chris Lu
1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00
Chris Lu
ea1169dc80 filer cipher: single chunk http POST and PUT and read 2020-03-07 06:08:08 -08:00
Chris Lu
7335e62199 volume: PUT also conditionally gzip compress 2020-03-01 23:37:02 -08:00
Chris Lu
151114ff59 volume: fix readonly status reporting 2020-03-01 17:57:37 -08:00
Chris Lu
f87f2045b3 delete old file first on windows 
fix https://github.com/chrislusf/seaweedfs/issues/1210
 2020-02-27 00:59:35 -08:00
Chris Lu
757c7d67ad avoid nil DataBackend 2020-02-27 00:07:25 -08:00
Chris Lu
430b5a49e2 unused code 2020-02-26 19:35:00 -08:00
Chris Lu
ed0acd1722 go fmt 2020-02-26 16:52:57 -08:00
Chris Lu
97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00