Chris Lu
828f6e9f4d
volume: auto add missing vif files
...
fix https://github.com/chrislusf/seaweedfs/issues/1878
2021-03-09 12:09:32 -08:00
qieqieplus
5b16820924
fix: collection stats won't update if all volumes expired at same iteration
2021-03-09 18:49:45 +08:00
Chris Lu
726edab054
avoid nil when closing an index
...
fix https://github.com/chrislusf/seaweedfs/issues/1870
2021-03-07 11:03:09 -08:00
Chris Lu
bdfed16d42
avoid nil exception
...
fix https://github.com/chrislusf/seaweedfs/issues/1869
2021-03-07 01:49:06 -08:00
Chris Lu
e9eddfb8a2
remove wrong writes
2021-03-06 14:26:27 -08:00
Chris Lu
38fc200e56
CRCWriter consistent with CRC
2021-03-06 14:26:27 -08:00
Chris Lu
400de380f4
volume server: support tcp direct put/get/delete
2021-03-05 02:29:38 -08:00
Chris Lu
c46d16b2f3
skip network error when remote volumes start
2021-03-01 01:20:06 -08:00
Chris Lu
6e43e8ce17
add logs for remote file loading
2021-03-01 00:48:59 -08:00
Chris Lu
015d16f43f
add vif file versions in case loading superblock fails
2021-03-01 00:48:30 -08:00
Chris Lu
d680676d45
skip already loaded backends
2021-03-01 00:47:03 -08:00
Chris Lu
540441fd38
go fmt
2021-02-28 20:34:14 -08:00
Chris Lu
2cca07b44c
cloud tier to non-AWS s3 gateways
2021-02-28 18:59:09 -08:00
Chris Lu
30b30b8fe0
volume.tier.move: passing non-empty disk type
2021-02-22 01:59:03 -08:00
Chris Lu
03c643aa6b
fix test
...
fileSize need to be divided by 8
2021-02-20 21:19:21 -08:00
Chris Lu
7635f6b9fa
disk file avoid file.Stat()
2021-02-20 20:06:06 -08:00
Chris Lu
a2383b3b12
Revert "Revert "avoid file.Stat()""
...
This reverts commit 7ef4c24f18
.
2021-02-20 12:42:09 -08:00
Chris Lu
7ef4c24f18
Revert "avoid file.Stat()"
...
This reverts commit 98c93ca465
.
2021-02-20 12:39:33 -08:00
Chris Lu
1ad3200094
skip seek() when index file writes
2021-02-20 12:39:25 -08:00
Chris Lu
98c93ca465
avoid file.Stat()
...
avoid one Syscall, but did not help on performance though
2021-02-20 09:44:17 -08:00
Chris Lu
a1210d1e8d
Revert "volume: avoid file.stat(), file.seek() if possible during writes"
...
This reverts commit c78409a598
.
2021-02-19 03:56:27 -08:00
Chris Lu
c78409a598
volume: avoid file.stat(), file.seek() if possible during writes
2021-02-18 23:22:54 -08:00
Chris Lu
73958e357d
add descriptive error if no free volumes
2021-02-18 19:10:20 -08:00
bingoohuang
352ac2f271
Merge remote-tracking branch 'origin/master'
2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34
use backticks instead of double quotes to avoid escaped additionally in regex
2021-02-18 14:05:28 +08:00
Chris Lu
3575d41009
go fmt
2021-02-17 20:57:08 -08:00
Chris Lu
7403cd43c5
only use "" for hdd to avoid two values for the same thing
2021-02-16 10:50:42 -08:00
Chris Lu
ebf320ec22
avoid divided by zero
2021-02-16 05:59:24 -08:00
Chris Lu
3fe628f04e
use hdd instead of empty string
2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab
this can compile now!!!
2021-02-16 02:47:02 -08:00
Chris Lu
7ce647f27e
support customizable disk type
2021-02-13 15:42:42 -08:00
Chris Lu
4bd8a692d8
disk type can be generic tags
2021-02-13 13:50:14 -08:00
Chris Lu
885ca34748
volume: fail fast if idx files are missing
...
fix https://github.com/chrislusf/seaweedfs/issues/1796
2021-02-11 00:44:40 -08:00
Chris Lu
821c46edf1
Merge branch 'master' into support_ssd_volume
2021-02-09 11:37:07 -08:00
bingoohuang
9e06ef66b9
fix renaming for 5bytes
2021-02-09 09:44:28 +08:00
Chris Lu
a833021132
fix refactoring left over
2021-02-07 23:03:03 -08:00
bingoohuang
7256902fb0
fix typo offset.ToAcutalOffset to offset.ToActualOffset
2021-02-07 12:11:51 +08:00
bingoohuang
94ea3bd3a5
renaming NeedleMapType to NeedleMapKind
2021-02-07 09:00:03 +08:00
Chris Lu
cfb9342a15
avoid concurrent map updates to viper
2021-01-12 02:28:13 -08:00
Chris Lu
039ad101b1
manifest file also need to detect compression
...
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753585046
2021-01-03 01:42:21 -08:00
Chris Lu
4f31c1bb94
go fmt
2020-12-22 02:34:08 -08:00
Chris Lu
9abf016af1
Merge branch 'master' into support_ssd_volume
2020-12-17 22:30:19 -08:00
Chris Lu
738c4fd203
fix logging when error
...
address https://github.com/chrislusf/seaweedfs/pull/1685#issuecomment-747845601
2020-12-17 19:37:46 -08:00
Chris Lu
daa8157fc2
Merge branch 'master' into support_ssd_volume
2020-12-17 13:05:20 -08:00
Chris Lu
e2076201d7
volume: avoid reprocessing the same volume
...
fix https://github.com/chrislusf/seaweedfs/issues/1682
2020-12-17 13:03:39 -08:00
Chris Lu
986cbdf7d9
Revert "Merge pull request #1683 from qieqieplus/master"
...
This reverts commit 8cb67952db
, reversing
changes made to 200e56215a
.
2020-12-17 12:46:42 -08:00
Chris Lu
f56e6d231e
Revert "Merge pull request #1683 from qieqieplus/master"
...
This reverts commit 8cb67952db
, reversing
changes made to 200e56215a
.
2020-12-17 12:46:20 -08:00
Chris Lu
1bf22c0b5b
go fmt
2020-12-16 09:14:05 -08:00
Chris Lu
1d88865869
passing disk type along
2020-12-16 09:10:14 -08:00
Chris Lu
23014b6810
Merge branch 'master' into support_ssd_volume
2020-12-16 08:26:51 -08:00
chenqieqie
45f902a9b7
smaller critical section
2020-12-16 20:23:51 +08:00
chenqieqie
4e58a4f24e
fix race condition when loading volumes concurrently
2020-12-16 18:49:10 +08:00
Chris Lu
2e8dba571b
adjust volume server UI
2020-12-14 00:51:57 -08:00
Chris Lu
933f2f4cfd
find location with matching disk type
2020-12-13 23:39:00 -08:00
Chris Lu
94525aa0fd
allocate volume by disk type
2020-12-13 23:08:21 -08:00
Chris Lu
f6a419c26c
disk type configurable for each folder
2020-12-13 22:49:56 -08:00
Chris Lu
0d2ec832e2
rename from volumeType to diskType
2020-12-13 11:59:32 -08:00
Chris Lu
d156c74ec0
volume server set volume type and heartbeat to the master
2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3
adding volume type
2020-12-13 00:58:58 -08:00
Chris Lu
ae655033ac
adjust logging
2020-12-11 16:57:53 -08:00
Chris Lu
d986c7196d
use append time instead of filer's own modification time
...
fix https://github.com/chrislusf/seaweedfs/issues/1669
2020-12-11 16:55:18 -08:00
Chris Lu
d171d9f988
volume: ensure the volume dat and idx files are always closed
...
fix https://github.com/chrislusf/seaweedfs/issues/1646
2020-12-04 21:50:26 -08:00
Chris Lu
cc839f935d
better unit detection
2020-12-03 19:23:59 -08:00
Chris Lu
b219ccfe68
Update volume_ttl_test.go
2020-12-03 12:55:44 -08:00
Chris Lu
a83b8e7b01
ttl fix: 3y was converted into 96m
2020-12-03 10:53:29 -08:00
Chris Lu
f3bb645018
file open error
2020-12-01 23:37:49 -08:00
Chris Lu
dc0bc48257
return file open error
2020-12-01 23:36:49 -08:00
Chris Lu
96c48bc8a8
fix test
2020-11-28 03:03:43 -08:00
Chris Lu
2c913dde04
volume: detect and drop volumes with disk IO error
...
from Jethro in slack:
is it possible to make the assign request a bit smarter? Currently I’m in the state that a disk failed but all assign request are being send to this volume. It would be cool if the master sees this and stopped using this volume.
e=HTTP(http://x:8089/913,045a782b63176edf ) not 200 but 500 Internal Server Error
Body={"size":740167,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"ee4381e202212ff3aee647704c036689"}
e=HTTP(http://x:8089/913,045a782c90240077 ) not 200 but 500 Internal Server Error
Body={"size":792779,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"c43463ccc11eb6eb2fc306f407a6a953"}
e=HTTP(http://x:8089/913,045a782e6b7901ea ) not 200 but 500 Internal Server Error
Body={"size":3962392,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"04c91198e9b276c81f11dbf189af5d28"}
2020-11-28 00:09:29 -08:00
Chris Lu
9ac4935f22
read from volume index file directly instead of open a separate file
...
fix https://github.com/chrislusf/seaweedfs/issues/1640
read from volume index file directly instead of open a separate file,
to ensure reading latest index entries.
2020-11-27 16:18:48 -08:00
Chris Lu
6d30b21b10
volume: add "-dir.idx" option for separate index storage
...
fix https://github.com/chrislusf/seaweedfs/issues/1265
2020-11-27 03:17:10 -08:00
Chris Lu
3c229eb677
refactor
2020-11-26 17:21:55 -08:00
Chris Lu
c0fb4a3766
add comments
2020-11-26 15:22:42 -08:00
Chris Lu
6783ae5f8f
rename
2020-11-26 15:19:43 -08:00
Chris Lu
97f706f35b
just refactoring
2020-11-26 15:08:05 -08:00
Chris Lu
b11449f955
pass in dir.idx parameter to DiskLocation
2020-11-26 14:59:03 -08:00
Chris Lu
0da7ecfd29
go fmt
2020-11-26 11:22:30 -08:00
Chris Lu
92f906b6fc
remove zstd
...
fix https://github.com/chrislusf/seaweedfs/issues/1629
2020-11-21 13:06:45 -08:00
Chris Lu
1dd3a6ac36
avoid verbose logs
2020-11-09 15:56:11 -08:00
Chris Lu
0dafcf1f5a
volume: detect max volume count changes based on disk usage
...
fix https://github.com/chrislusf/seaweedfs/issues/1594
2020-11-03 14:43:17 -08:00
Chris Lu
de86945aeb
go fmt
2020-10-31 16:45:38 -07:00
Chris Lu
9708df47d1
ensure offset in the index file is positive
2020-10-31 15:50:01 -07:00
Konstantin Lebedev
46303c36bf
When the volume server is stopped, the master server immediately sees the deletion of volumes
2020-10-31 02:48:25 +05:00
Chris Lu
6560ac6466
volume loading: trim out unreachable idx file content
...
fix https://github.com/chrislusf/seaweedfs/issues/1583
2020-10-29 22:25:23 -07:00
Konstantin Lebedev
1ad1b8c4f6
collection Volume ReadOnly Count with detailed status
2020-10-28 17:38:26 +05:00
Chris Lu
744c5594da
volume: automatically trim out unreachable entries
2020-10-28 01:14:39 -07:00
Chris Lu
3e925faddd
trim out name extension
2020-10-28 01:14:05 -07:00
shibinbin
a67eb1afa1
fix: restart volumeserver sometime occurred out of memory error
2020-10-28 12:03:06 +08:00
Chris Lu
53c3aad875
volume: add a note file to avoid incomplete volume files
...
fix https://github.com/chrislusf/seaweedfs/issues/1567
2020-10-27 15:56:49 -07:00
Chris Lu
06c15ab35c
volume: add special handling for .dat larger than 32GB
2020-10-27 13:11:56 -07:00
Konstantin Lebedev
2fb1fce8a0
avoid old values when missing read only flags
2020-10-25 16:21:40 +05:00
Chris Lu
9104cfa744
reduce locks
2020-10-24 19:40:35 -07:00
Chris Lu
e9d40b80b2
less verbose logs
2020-10-24 09:42:54 -07:00
Chris Lu
575d7952a1
add available resource stats
...
fix https://github.com/chrislusf/seaweedfs/issues/1555
2020-10-22 09:13:47 -07:00
Chris Lu
4ff2ceee33
UI fix on rendering EC volumes
...
addressing UI problem with https://github.com/chrislusf/seaweedfs/issues/1551
2020-10-21 22:05:58 -07:00
Konstantin Lebedev
dc2e13092d
add number of read only volumes metric
2020-10-15 15:48:40 +05:00
Chris Lu
bd8b9b0c2e
some changes to prepare fix after failed reads
2020-10-08 18:33:06 -07:00
Chris Lu
d522df0ecd
adjust log level
2020-10-04 01:35:32 -07:00
Chris Lu
a1c01d716b
volume: avoid deadlock when deleting volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1501
2020-10-01 07:10:03 -07:00
Chris Lu
103fafe00b
weed export: print out [start,stop) content range
2020-09-14 22:57:23 -07:00
Chris Lu
f2723c1bc8
do not idx file format
...
revert c9ab8d05fa
2020-09-12 12:42:36 -07:00
Chris Lu
446e476a11
go fmt
2020-09-12 04:08:03 -07:00
Chris Lu
ea26a98753
volume: validate volume correctness if last entry is a deletion
2020-09-12 04:07:04 -07:00
Chris Lu
9d4bdfcfdf
fix volume integrity checking
2020-09-11 11:34:10 -07:00
Chris Lu
401ccf1509
fix test
2020-09-10 15:13:23 -07:00
Chris Lu
c9ab8d05fa
fixes for reading deleted fid
2020-09-10 14:42:52 -07:00
James Hartig
8e54e34576
volume: Don't unmount before deleting volume in copy
...
If we unmount first and then delete, the delete fails because the volume
was unmounted. Delete ends up doing the same thing as the unmount anyways.
2020-09-01 22:00:07 -04:00
Chris Lu
38e06d783d
volume: check disk space before compaction
...
fix https://github.com/chrislusf/seaweedfs/issues/1440
2020-08-31 18:10:53 -07:00
Chris Lu
dbb10e0f0b
testing with more than 30days
2020-08-31 17:59:29 -07:00
Chris Lu
99ecf63276
go fmt
2020-08-29 22:28:33 -07:00
Chris Lu
ab759f0ec2
erasure coding: fix EC error if multiple disks are configured in one volume server
2020-08-26 09:16:58 -07:00
Chris Lu
d1cf39f180
fix logging
2020-08-26 08:55:15 -07:00
James Hartig
3ccfa4c6ad
Added VolumeMarkWritable and VolumeStatus grpc methods
...
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
2020-08-19 11:42:56 -04:00
Chris Lu
3b4b1d4a77
fix tests
2020-08-19 01:37:56 -07:00
Chris Lu
ed4b43b419
adjust logs
2020-08-18 23:42:09 -07:00
Chris Lu
fe01191b5b
support read option readDeleted=true
2020-08-18 19:22:16 -07:00
Chris Lu
6ccd7f0a4d
refactoring
2020-08-18 18:01:37 -07:00
Chris Lu
c026eb0592
refactoring
2020-08-18 17:39:29 -07:00
Chris Lu
7e91ae592c
pass in option to read deleted entries
...
not working yet
2020-08-18 17:37:26 -07:00
Chris Lu
ee11d98650
refactoring
2020-08-18 17:35:19 -07:00
Chris Lu
6a92f0bc7a
refactoring to typed Size
...
Go is amazing with refactoring!
2020-08-18 17:04:28 -07:00
Chris Lu
627b081b67
adjust logs
2020-08-16 16:32:22 -07:00
Chris Lu
c03bb180eb
fix error reporting
2020-08-14 00:44:02 -07:00
Chris Lu
ab6e5c0dc4
adjust error message
2020-08-08 10:18:32 -07:00
Chris Lu
20e2ac1add
filer: store md5 metadata for files uploaded by filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1412
2020-08-06 10:04:17 -07:00
Chris Lu
4ecfa9879d
volume: report Content-MD5 in response header
2020-08-06 05:22:53 -07:00
Chris Lu
cbd80253e3
better needle id format
2020-08-04 22:09:07 -07:00
Chris Lu
f89186a3cf
better logging
2020-08-01 11:19:57 -07:00
Chris Lu
943fa4ea0c
fix compilation error
2020-07-22 16:11:17 -07:00
Chris Lu
49446af516
simplify
2020-07-22 13:43:54 -07:00
James Hartig
0be3bb4a87
idx: Switch WalkIndexFile to accept generic io.ReaderAt
...
This is necessary to use it with in-memory idx files.
2020-07-22 15:01:16 -04:00
Chris Lu
97d97f3528
go code can read and write chunk manifest
2020-07-19 17:59:43 -07:00
Chris Lu
f43146b237
resolve directories if containing home directory
2020-07-16 22:50:14 -07:00
Chris Lu
ddec7b2bb9
go fmt
2020-07-14 07:34:16 -07:00
Chris Lu
f2e8ad6241
refactoring
2020-07-11 06:35:54 -07:00
李远军
6608cb5f43
Revert "remove fixJpgOrientation"
...
This reverts commit de5ca9b2
2020-07-10 10:08:36 +08:00
Chris Lu
4bed1663e2
adjust log level
2020-07-07 23:17:31 -07:00
Chris Lu
4d1484628a
refactoring
2020-07-03 16:41:30 -07:00
Chris Lu
faa5c2e89a
refactoring
2020-07-03 16:34:31 -07:00
Chris Lu
20d9cfae86
volume: following md5 base64 encoding
...
fix https://github.com/chrislusf/seaweedfs/issues/1374
2020-06-28 07:25:50 -07:00
Chris Lu
212c64241b
checking ecx file
2020-06-25 16:36:06 -07:00
Chris Lu
9fd7cdadf1
fix
2020-06-25 10:45:34 -07:00
Chris Lu
3b638d3994
add more ec encoding logging
2020-06-25 09:43:38 -07:00
Chris Lu
32a1f22730
more logs
2020-06-24 22:07:53 -07:00
Chris Lu
2ff37ccdbd
testing parsing the upload
...
The mime type is always the value passed in.
Compress or not depends on the content detection, file name extension, and compression ratio.
If the content is already compressed, need to know the content size.
2020-06-24 11:39:12 -07:00
Chris Lu
c21f4ebfee
Merge pull request #1371 from Kimbsen/content_md5_validation
...
Optional md5 validation of uploads
2020-06-24 11:33:44 -07:00
Kimbsen
4eeab2a379
remove unused imports
2020-06-24 13:37:00 +02:00
Kimbsen
ffddecebef
do md5 validation AFTER decompression
2020-06-24 13:35:13 +02:00
Chris Lu
fe60db404a
prepare for zstd
2020-06-23 09:12:02 -07:00
Kimbsen
6b1e93ba0b
check request header not part header for content-encoding
2020-06-23 17:04:24 +02:00
Kimbsen
38626cb584
don't forget the last few bytes
2020-06-23 16:53:49 +02:00
Kimbsen
8d1f6cd567
added optional md5 verification of uploaded data
2020-06-23 16:24:11 +02:00
Chris Lu
de5ca9b258
remove fixJpgOrientation
2020-06-20 12:50:40 -07:00
Chris Lu
3080c197e3
rename UnCompressData to DecompressData
2020-06-20 08:16:16 -07:00
Chris Lu
e912fd15e3
renaming
2020-06-19 22:45:27 -07:00
Chris Lu
16fe132a20
detect mime type before replicating to other volume servers
2020-06-19 22:11:36 -07:00
Chris Lu
ae1994cbc1
erasure coding: fix cases where there are no .ecj files
2020-06-18 09:52:35 -07:00
Chris Lu
7ccc31853c
hard limit volume file size
2020-06-16 00:27:48 -07:00
Chris Lu
bdbe5ddbe9
volume server: add a fix for dead lock
...
https://github.com/chrislusf/seaweedfs/issues/1353
2020-06-08 08:12:59 -07:00
Chris Lu
45a9d852d3
go fmt
2020-06-05 15:27:10 -07:00
Evgenii Kozlov
2410ffbd79
Name fix
2020-06-05 19:00:38 +03:00
Evgenii Kozlov
0e0db70f55
Set volumes ReadOnly if low free disk space
2020-06-05 18:18:15 +03:00
Chris Lu
ef2b3a0801
leveldb recover error handling
2020-05-26 00:03:44 -07:00
Chris Lu
e5a0787653
filer: leveldb store add auto recovery, report correct error
...
fix https://github.com/chrislusf/seaweedfs/issues/1326
2020-05-22 10:54:42 -07:00
Chris Lu
7ff2b0121d
specify constant data type to uint64 instead of default int
...
fix https://github.com/chrislusf/seaweedfs/issues/1324
2020-05-19 07:55:23 -07:00
Chris Lu
11b33c0760
fix compilation problem
2020-05-18 10:53:31 -07:00
zhangsong
b161baa3f0
fix volume vacuum test unit
2020-05-07 19:58:06 +08:00
zhangsong
3703c7e176
write request keep compatible with before
2020-05-06 21:35:32 +08:00
zhangsong
f9e8702bb4
use async write to persistent file to disk - part1
2020-05-04 17:39:44 +08:00
Chris Lu
6190fd665d
printout error
2020-04-27 12:41:31 -07:00
Chris Lu
bafa95045b
volume: deletion checks all disk locations
...
fix https://github.com/chrislusf/seaweedfs/issues/1283
2020-04-21 14:49:58 -07:00
Chris Lu
7764e0465c
refactoring
2020-04-12 21:00:55 -07:00
Chris Lu
e4af63a721
volume server: accept fsync=true in write requests
2020-04-11 21:39:16 -07:00
Chris Lu
c8ca234773
refactoring
2020-04-11 14:27:25 -07:00
Chris Lu
df9d538044
rename function
2020-04-11 14:19:44 -07:00
Chris Lu
9fa065f600
typo in logs
2020-04-09 23:43:09 -07:00
Chris Lu
e63a79ade8
better handle lock in case of exception
2020-03-24 18:41:25 -07:00
Chris Lu
3137777d83
volume: automatically detect max volume count
2020-03-22 16:21:42 -07:00
Chris Lu
cbfe31a9a8
idx file sync before compaction
2020-03-20 23:38:46 -07:00
Chris Lu
81797a059a
volume: sync volume file right before compaction
...
fix https://github.com/chrislusf/seaweedfs/issues/1237
2020-03-19 23:54:52 -07:00
Chris Lu
709f231e23
tiered storage: add s3 endpoint for private s3 implementation
...
fix https://github.com/chrislusf/seaweedfs/issues/1238
2020-03-19 21:13:56 -07:00
Chris Lu
b964bbab3d
fix compilation
2020-03-17 10:01:24 -07:00
Chris Lu
c3cb6fa1d7
volume: compaction can cause readonly volumes
...
address https://github.com/chrislusf/seaweedfs/issues/1233
2020-03-17 09:43:57 -07:00
Chris Lu
22400c6633
consistent gzip logic
...
local store gzip same as replicated writes
2020-03-15 04:15:40 -07:00
Chris Lu
560df51def
refactoring
2020-03-15 03:11:26 -07:00
Chris Lu
d439d83772
volume: follow compactionBytePerSecond
...
related to https://github.com/chrislusf/seaweedfs/issues/1108
2020-03-11 10:32:17 -07:00
Chris Lu
d3e1ef64b3
ensure error checking on windows
2020-03-10 13:33:26 -07:00
Chris Lu
88a110e67e
release file handle
2020-03-10 13:32:22 -07:00
Chris Lu
43e6261579
volume: fix memory leak
2020-03-10 13:31:23 -07:00
Chris Lu
0871d2cff0
volume: fix memory leak during compaction
...
fix https://github.com/chrislusf/seaweedfs/issues/1222
2020-03-09 22:29:02 -07:00
Chris Lu
89eb05b50f
filer: support TTL for all filer stores
2020-03-09 01:02:01 -07:00
Chris Lu
9b3109a5d8
filer: processing all response headers, no pass through to volume server
...
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
2020-03-08 15:42:44 -07:00
Chris Lu
1ae83c2938
go fmt
2020-03-07 06:12:57 -08:00
Chris Lu
ea1169dc80
filer cipher: single chunk http POST and PUT and read
2020-03-07 06:08:08 -08:00
Chris Lu
7335e62199
volume: PUT also conditionally gzip compress
2020-03-01 23:37:02 -08:00
Chris Lu
151114ff59
volume: fix readonly status reporting
2020-03-01 17:57:37 -08:00
Chris Lu
f87f2045b3
delete old file first on windows
...
fix https://github.com/chrislusf/seaweedfs/issues/1210
2020-02-27 00:59:35 -08:00
Chris Lu
757c7d67ad
avoid nil DataBackend
2020-02-27 00:07:25 -08:00
Chris Lu
430b5a49e2
unused code
2020-02-26 19:35:00 -08:00
Chris Lu
ed0acd1722
go fmt
2020-02-26 16:52:57 -08:00
Chris Lu
97ab8a1976
remove ctx if possible
2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9
avoid reusing context object
...
fix https://github.com/chrislusf/seaweedfs/issues/1182
2020-02-25 21:50:12 -08:00
panyc16
7b3764fd9e
make loadExistingVolume() return bool
2020-02-13 15:19:00 +08:00
Chris Lu
8d94564f41
refactor
2020-02-04 21:16:34 -08:00
Chris Lu
c1288e9eb4
volume: sdx generation uses memdb instead of compactMap
...
fix https://github.com/chrislusf/seaweedfs/issues/1194
2020-02-04 21:12:09 -08:00
Chris Lu
08e4702542
sync before closing index file
...
merge from 0181f87d91
2020-02-04 10:38:12 -08:00
Chris Lu
3a35632d58
sync before closing file
...
merge changes from 1747fc2d52
2020-02-04 10:37:14 -08:00
Chris Lu
40ae533fa3
shell: add volume.configure.replication to change replication for a volume
...
fix https://github.com/chrislusf/seaweedfs/issues/1192
2020-02-02 15:37:23 -08:00
Chris Lu
d335f04de6
support env variables to overwrite toml file
2020-01-29 09:09:55 -08:00
Chris Lu
72a64a5cf8
use the same context object in order to retry
2020-01-26 14:42:11 -08:00
Chris Lu
b6e6ca8595
adjust logs
2020-01-24 20:06:58 -08:00
Chris Lu
c8b2dac6c1
volume: avoid sharing volume dat file handle
...
possibly help on https://github.com/chrislusf/seaweedfs/issues/1184
2020-01-21 21:18:01 -08:00
Chris Lu
f8a20ef35e
add the old way to compact as a comment
2020-01-08 09:45:42 -08:00
Chris Lu
288baf37fd
saving .vif files correctly
2020-01-08 09:45:26 -08:00
Chris Lu
acf7ca7b93
volume: fix compaction
2020-01-08 09:45:03 -08:00
Chris Lu
943f4986ef
fix possible nil
2020-01-08 08:49:18 -08:00
Chris Lu
9995d3bcb5
remove println
2020-01-07 21:49:28 -08:00
Chris Lu
3eafec4b29
volume: add option to limit file size
2020-01-03 00:37:26 -08:00
Chris Lu
672868b460
always create .vif file
2019-12-28 21:52:06 -08:00
Chris Lu
b6f7033478
go fmt
2019-12-28 21:37:29 -08:00
Chris Lu
9134092a8d
endure creating .vif files
2019-12-28 21:36:15 -08:00
Chris Lu
0cc8347a3a
go fmt
2019-12-28 21:13:10 -08:00
Chris Lu
37b64a50b4
ec: generate and copy .vif file
2019-12-28 12:44:59 -08:00
Chris Lu
c06f7eb48a
load volume info from .vif file, use superblock as a backup
2019-12-28 12:28:58 -08:00
Chris Lu
f4a74e03d1
refactoring: separating .vif from tier file loading
2019-12-28 11:35:27 -08:00
Chris Lu
2000284435
rename volume tier info to volume info
2019-12-28 11:21:49 -08:00
Chris Lu
e8b357fd53
rename from .tier to .vif
2019-12-28 11:17:39 -08:00
Chris Lu
48d28d3eb2
tier: support remote file attributes and remember the file extension
2019-12-25 21:37:24 -08:00
Chris Lu
1346437d71
Merge pull request #1171 from stlpmo-jn/redundant_type_conversion
...
remove the redundant type conversion
2019-12-25 17:02:05 -08:00
Chris Lu
35393b4a02
cleanly destroy remote files
2019-12-25 16:17:58 -08:00
Chris Lu
eecad6062a
go fmt
2019-12-25 09:56:42 -08:00
Chris Lu
d960b3474a
tier storage: support downloading the remote dat files
2019-12-25 09:53:13 -08:00
stlpmo
f3de4b6c18
remove the redundant type conversion
2019-12-25 10:13:45 +08:00
Chris Lu
9ff72f616a
go fmt
2019-12-24 14:56:16 -08:00
Chris Lu
efd2f50ede
compaction changed to .idx based deletion
2019-12-24 14:55:50 -08:00
Chris Lu
d8b39fe92a
testing
2019-12-24 11:29:26 -08:00
Chris Lu
f42b5bd0f5
rename .sdb to .sdx
2019-12-24 10:19:12 -08:00
Chris Lu
abffe857a1
change btree map to in memory level db
2019-12-24 10:18:56 -08:00
Chris Lu
72a561ab7c
refactoring
2019-12-24 08:13:01 -08:00
Chris Lu
409a3fe41f
Merge pull request #1168 from stlpmo-jn/optimize_DeleteCollectionFromDiskLocation
...
decouple the volume.Destroy() from the operation of unmountVolume()
2019-12-24 07:08:04 -08:00
stlpmo
38e4b79125
decouple the volume.Destroy() from the operation of unmountVolume()
2019-12-24 17:20:34 +08:00
stlpmo
f300c35266
resolve issue : when the host disk full, the volume server will core dump
2019-12-24 16:36:15 +08:00
Chris Lu
a18f62bbe7
only copy required shards
2019-12-23 18:06:13 -08:00
Chris Lu
8a2d529759
go fmt
2019-12-23 12:48:53 -08:00
Chris Lu
09ca936c78
shell: add ec.decode command
2019-12-23 12:48:20 -08:00
Chris Lu
ab966d7192
refactoring variable name
2019-12-19 00:44:46 -08:00
Chris Lu
f61de28c69
volume: add deletion capability for previously readonly volumes
2019-12-19 00:42:46 -08:00
Chris Lu
deb03ecfa3
remove duplicated checking
...
already checked in store.go
2019-12-18 23:59:53 -08:00
Chris Lu
ef3ae3cd41
clean up sorted index file
2019-12-18 20:59:15 -08:00
Chris Lu
39810a9951
tiered volume is read only
2019-12-18 01:26:22 -08:00
Chris Lu
58f88e530c
volume: use sorted index map for readonly volumes
2019-12-18 01:21:21 -08:00
Chris Lu
356bd1b629
adjust text
2019-12-10 22:09:29 -08:00
Chris Lu
10bd3c6b4b
refactoring
2019-12-08 19:44:16 -08:00
Chris Lu
2b8e20f122
remove unused functions
2019-12-08 18:33:25 -08:00
Chris Lu
f38f90b7ea
cleanly remove .tier volume file
2019-12-06 22:19:47 -08:00
Chris Lu
6827cabfbb
volume: dataFileAccessLock change to RW Lock
2019-12-06 06:59:57 -08:00
Chris Lu
e426bd541e
grow volumes on volume servers with slots freed by cloud storage
2019-12-03 21:36:42 -08:00
Chris Lu
7ae8b1cc86
show volume info in volume.list in weed shell
2019-12-02 23:38:56 -08:00
Chris Lu
1becbce657
display remote volumes on volume server ui page
2019-12-02 23:23:54 -08:00
Chris Lu
70648d35ad
go fmt
2019-12-02 20:49:58 -08:00
Chris Lu
6383b45bd0
add lock variable
2019-12-02 20:49:50 -08:00
Chris Lu
caae543a9f
fix test
2019-12-02 15:54:24 -08:00
Chris Lu
ec8de250e2
tiered storage: can copy to s3, read from s3
...
master not aware tiered volume yet, file assigning is not working yet
2019-12-02 15:08:31 -08:00
Chris Lu
0da7b894cc
pass backend config from master to volume servers
2019-11-29 01:05:09 -08:00
Chris Lu
61bc1d6ffd
tweaking
2019-11-28 18:47:51 -08:00
Chris Lu
f60154f330
master load backend storage config from master.toml
2019-11-28 18:33:18 -08:00
Chris Lu
ee2072dff9
volume: avoid inifinite loop reading ec volume info if error happens
2019-11-27 23:07:53 -08:00
Chris Lu
0f9ba84274
s3
2019-11-27 03:09:45 -08:00
Chris Lu
e4c63ff2f8
go fmt
2019-11-22 14:51:09 -08:00
Chris Lu
9711a6ffaa
WIP
2019-11-18 19:24:37 -08:00
chenwanli
295f3beed5
rm ldb and bdb directory
2019-11-19 09:35:06 +08:00
Chris Lu
c6e8225a20
adjust memory mapped file size
...
related to https://github.com/chrislusf/seaweedfs/pull/1087
2019-11-12 21:05:48 -08:00
Lei Liu
46755ea1e1
fix master maintenance logic
...
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-11-12 14:46:10 +08:00
Chris Lu
85f8649320
refactor memory mapped file into backend storage
2019-11-09 00:10:59 -08:00
Chris Lu
c5c1d83d91
Merge branch 'master' into refactoring_dat_backend
2019-11-08 22:54:41 -08:00
Chris Lu
c34ffed43f
go fmt
2019-11-08 22:47:50 -08:00
Chris Lu
84c503c6a7
adjust ec reading log level
2019-11-08 22:41:02 -08:00
zhangsong
1dd101f782
use read lock to avoid io hang during heartbeat
2019-11-08 20:27:55 +08:00
Chris Lu
db30a46050
skip memory mapped file when compacting
2019-10-29 23:34:38 -07:00
Chris Lu
5b950c735e
Merge branch 'master' into refactoring_dat_backend
2019-10-29 23:18:41 -07:00
Chris Lu
57e441d67b
fix compaction logic
2019-10-29 23:18:01 -07:00
Chris Lu
b7156291a8
adjust variable names
2019-10-29 23:16:43 -07:00
Chris Lu
4b5ba4927f
adjust NewDiskFile() paramaters
2019-10-29 22:37:36 -07:00
Chris Lu
cd7ce720c7
fix test
2019-10-29 00:37:15 -07:00
Chris Lu
19b6a16003
changed from os.file to backend.DataStorageBackend
2019-10-29 00:35:16 -07:00
Chris Lu
7a51a9a582
refactor: simplifying to ReadAt() and WriteAt()
2019-10-25 00:11:25 -07:00
Chris Lu
46ed2ca902
refactoring
2019-10-24 23:41:32 -07:00
Chris Lu
f937933b3d
refactoring
2019-10-23 22:25:53 -07:00
Chris Lu
3c865ee39b
fix missing needle header read
2019-10-22 00:57:47 -07:00
Chris Lu
fc412e428b
refactor ScanVolumeFileFrom()
2019-10-22 00:50:30 -07:00
Chris Lu
c9a183eb69
refactor memory map related code
2019-10-22 00:49:42 -07:00
Chris Lu
fec07c829d
go fmt
2019-10-21 23:03:48 -07:00
Chris Lu
faec9076a4
adjust parameter names
2019-10-21 22:57:01 -07:00
j.laycock
2e2fe00dbd
Comma, no space?
2019-10-18 11:31:25 +01:00
j.laycock
248f3be6e3
using a space instead of a comma to hopefully fix the build!
2019-10-18 11:23:02 +01:00
j.laycock
2c455841ea
Make volumeCreate more consistent between all 3 implementations.
2019-10-18 11:01:45 +01:00
j.laycock
b0ddad6889
Fix volume_create breaking the build
2019-10-18 10:32:07 +01:00
joeslay
d53aee179b
Merge pull request #10 from chrislusf/master
...
merge seaweed master
2019-10-14 16:03:40 +01:00
Chris Lu
87e5a02a99
fix logic error
2019-10-09 00:07:18 -07:00
Chris Lu
09874f0d16
volume: return error if superblock is not initialized
...
fix https://github.com/chrislusf/seaweedfs/issues/1079
2019-10-09 00:02:50 -07:00