Chris Lu
50be19d23e
refactor
2021-05-09 15:33:01 -07:00
Chris Lu
55e060cf61
refactor
2021-05-09 15:28:54 -07:00
Chris Lu
735e65be17
refactor
2021-05-09 15:22:38 -07:00
Chris Lu
93e84a12f2
refactor
2021-05-09 15:15:18 -07:00
Chris Lu
3942e3b2ef
a better fix
2021-05-09 01:42:19 -07:00
Chris Lu
957e1a1bc1
fuse: important: if filer -filer.path is not root, directory listing will fail
2021-05-09 00:32:21 -07:00
Chris Lu
7ca75347ec
minor
2021-05-07 21:56:45 -07:00
Chris Lu
8f8738867f
add retry to assign volume
...
fix https://github.com/chrislusf/seaweedfs/issues/2056
2021-05-07 07:29:26 -07:00
Chris Lu
007401f3a0
remove duplicated code
2021-05-07 07:14:24 -07:00
Chris Lu
aaad4b578b
Merge pull request #2049 from qieqieplus/async-assign
...
ahead of time volume assignment
2021-05-06 10:00:25 -07:00
qieqieplus
c4d32f6937
ahead of time volume assignment
2021-05-06 18:55:44 +08:00
Chris Lu
55a8f57381
go fmt
2021-05-06 03:37:51 -07:00
Chris Lu
5753749c90
remove verbose logs
2021-05-06 03:34:34 -07:00
Chris Lu
38f411219a
mount: skip local chunk cache if opened write only
2021-05-06 03:31:40 -07:00
Chris Lu
c899bdf063
a little optimization
2021-05-06 03:03:00 -07:00
Chris Lu
3eb336e0b0
report error only for the first multipart upload
...
the glog.Errorf would always print for s3 multipart uploads
2021-05-06 01:57:54 -07:00
Chris Lu
76c48ffe27
optional parallel copy ec shards
...
fix https://github.com/chrislusf/seaweedfs/issues/2048
2021-05-06 01:53:35 -07:00
Chris Lu
ac71117ee6
revert PR #1903 avoid http error: superfluous response.WriteHeader
2021-05-05 15:11:39 -07:00
Chris Lu
24efa31e49
Merge pull request #2045 from qieqieplus/fix-vacuum-commit
2021-05-05 07:41:38 -07:00
qieqieplus
ac26080bd2
fix concurrent vacuum & delete panic
2021-05-05 17:54:50 +08:00
Chris Lu
e24ba2aadc
filer: delete specific tags
...
fix https://github.com/chrislusf/seaweedfs/issues/2041
2021-05-02 21:53:43 -07:00
Chris Lu
e87e6ef33c
s3: return 404 if bucket does not exist
...
fix https://github.com/chrislusf/seaweedfs/issues/2039
2021-05-02 21:30:37 -07:00
Chris Lu
a4bb37a5fe
add debug info
2021-05-01 16:10:26 -07:00
Chris Lu
2e56407c6b
fix visited checking
2021-05-01 16:09:29 -07:00
Chris Lu
c48ef78670
2.43
2021-05-01 00:39:04 -07:00
Chris Lu
3a86d4dbfd
mount: fix directory invalidation
...
fix https://github.com/chrislusf/seaweedfs/issues/2038
2021-04-30 22:51:06 -07:00
Chris Lu
d74cdf0115
Revert "Merge pull request #2027 from bingoohuang/master"
...
Need to revert because docker image build failed. The docker apk package only has go 1.15.
2021-04-30 03:36:15 -07:00
Chris Lu
84312e6799
2.42
2021-04-30 03:14:07 -07:00
Chris Lu
fa0d973113
Merge pull request #2037 from utsl42/reader_at
...
make reader_at handle random reads more efficiently for FUSE
2021-04-28 17:31:46 -07:00
Nathan Hawkins
042de9359c
make reader_at handle random reads more efficiently for FUSE
2021-04-28 19:13:37 -04:00
Chris Lu
a26a37dfa3
fix compilation
...
fix related to #2032
2021-04-28 13:36:53 -07:00
Chris Lu
ef94ff6837
Merge pull request #2035 from kmlebedev/fix_chunks_etag
...
fix aws style Etag for chunks
2021-04-28 10:37:18 -07:00
Chris Lu
8ae3ea4b1d
Merge pull request #2034 from kmlebedev/iam_listkeys_filter
...
iam ListAccessKeys filtred by username
2021-04-28 10:34:40 -07:00
Chris Lu
9dca75aea8
Merge pull request #2032 from tobiasmuehl/patch-2
...
Detect rar archives by mime type
2021-04-28 10:31:19 -07:00
Konstantin Lebedev
c2269123d3
fix aws style Etag for chunks
2021-04-28 22:28:05 +05:00
Konstantin Lebedev
39f636c682
iam ListAccessKeys filtred by username
2021-04-28 17:15:22 +05:00
Tobias Mühl
a8864e2abd
Detect rar archives by mime type
...
RAR archives might not have .rar extension, see [Wikipedia](https://en.wikipedia.org/wiki/RAR_(file_format) )
2021-04-28 15:54:19 +07:00
Tobias Mühl
12a7e87007
Do not compress brotli archives
2021-04-28 15:51:49 +07:00
Konstantin Lebedev
a48785c7df
auth use bucket wild cards
2021-04-27 21:45:40 +05:00
bingoohuang
7a9d27fce8
promote to go:embed instead of github.com/rakyll/statik
2021-04-27 17:22:24 +08:00
bingoohuang
cf552417a7
minFreeSpace refactored
2021-04-27 10:37:24 +08:00
bingoohuang
31f1cdeac2
minFreeSpace argument allows size like 10GiB
2021-04-26 18:48:34 +08:00
Chris Lu
86185262bb
2.41
2021-04-24 16:54:36 -07:00
Chris Lu
79f2e780c1
ensure name pattern checking is case sensitive
2021-04-24 11:51:23 -07:00
Chris Lu
ddc8643ee0
filer: directory listing adds namePatternExclude
...
fix https://github.com/chrislusf/seaweedfs/issues/2023
2021-04-24 11:49:03 -07:00
Chris Lu
f0ad172e80
shell: show which server holds the lock
...
fix https://github.com/chrislusf/seaweedfs/issues/1983
2021-04-22 23:56:35 -07:00
Chris Lu
89eb9f6e70
clean up .uploads directory in mysql and postgres tables
...
fix https://github.com/chrislusf/seaweedfs/issues/1957
When no uploads are running, you can run this SQL to clean up.
delete from <bucket_name> where directory like '/.uploads/%'
2021-04-22 23:23:23 -07:00
Chris Lu
46ef1811a1
correct help message
2021-04-22 22:26:38 -07:00
Chris Lu
11c120c040
master UI adds volume size limit
2021-04-22 14:22:48 -07:00
Chris Lu
6e5df901e4
adjust package names
2021-04-22 14:22:48 -07:00
Chris Lu
ca998328c2
do not add new volumes when below minFreeSpacePercent
...
fix https://github.com/chrislusf/seaweedfs/issues/2017
2021-04-21 23:11:11 -07:00
Chris Lu
cd7bf1a72a
filer.copy copy empty folders
...
fix https://github.com/chrislusf/seaweedfs/issues/2016
2021-04-21 11:17:43 -07:00
Chris Lu
a8114da02d
avoid thundering herd effect
...
transient errors may cause thundering herd effect to all trying to recover from remove ec shards
2021-04-21 10:17:12 -07:00
Chris Lu
ae74d8f02a
fix error message
...
related to https://github.com/chrislusf/seaweedfs/issues/2012
2021-04-21 01:40:16 -07:00
Chris Lu
11c405fc85
ensure file handles are released
2021-04-20 19:56:55 -07:00
liuxiaobo
c31c5e829c
fix path-specific filer store comment error
2021-04-20 10:08:58 +08:00
Chris Lu
83cf94ad2d
delay new file creation unless file is opened exclusively
2021-04-19 10:58:25 -07:00
Chris Lu
e983f91b03
2.40
2021-04-18 13:58:01 -07:00
Chris Lu
d1c813c470
let the fuse library manage directory id
...
otherwise, on mac, during large directory deletion, if some ReaDirAll happens, the lib seems confused about the directories, and some child directories are not deleted.
2021-04-18 13:07:28 -07:00
Chris Lu
372872ebbf
set root node inode number
2021-04-18 13:07:28 -07:00
Chris Lu
e332da4837
set inode value
2021-04-18 13:07:28 -07:00
Chris Lu
2acf6be24e
resend the http request if connection is stale
2021-04-18 13:07:28 -07:00
Chris Lu
6cbd786db9
correctly runs git clone
2021-04-18 13:07:28 -07:00
Chris Lu
d41e6826d3
adjust logging
2021-04-18 13:06:38 -07:00
Chris Lu
d9a2a7f1c4
WIP
...
no memory issue
if some directory is removed, it may have this error
$ rm -Rf ~/tmp/m2/s1
rm: fts_read: Device not configured
2021-04-18 13:06:38 -07:00
Chris Lu
54410ca955
cleaner way to set readonly
2021-04-18 10:02:02 -07:00
Konstantin Lebedev
198688c717
revert volume etag
2021-04-16 23:22:31 +05:00
Chris Lu
c83ab91e2e
remove unused variable
2021-04-16 10:34:02 -07:00
Chris Lu
3074e9b428
ensure consistent inode value
2021-04-15 22:42:24 -07:00
Chris Lu
b971317a16
avoid possible corrupted file names
2021-04-15 11:41:34 -07:00
Chris Lu
283d703d50
adjust text
2021-04-15 11:29:58 -07:00
Chris Lu
609e228578
avoid forward slash in file names
2021-04-15 10:53:04 -07:00
Chris Lu
ba92f2e714
add node.selectedVolumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1990
2021-04-15 03:19:28 -07:00
Chris Lu
16c0304416
ensure to delete on filer also
2021-04-15 02:29:04 -07:00
Chris Lu
217e0f9066
mount: remove folder recursively
2021-04-15 01:51:10 -07:00
Chris Lu
3e669e6d7b
mostly refactoring, add some error handling
2021-04-14 23:33:37 -07:00
Chris Lu
07f712c83f
fix typo
2021-04-14 23:21:38 -07:00
Chris Lu
36c79de3f4
fuse mount: dir ReadDirAll avoid extra conversion to filer_pb.Entry
2021-04-14 23:21:24 -07:00
Chris Lu
e41766feb6
fuse mount: dir lookup avoids extra conversion to filer_pb.Entry object
2021-04-14 22:38:34 -07:00
Chris Lu
1adc8f86ea
lighten up File object
...
file.entry only exists when file.isOpen
2021-04-14 20:49:15 -07:00
Chris Lu
6bc09b18c4
truncate is a bit faster to reuse the storage
2021-04-14 20:26:56 -07:00
Chris Lu
c04b7e106f
mount: remove entry from Dir object
2021-04-14 20:26:13 -07:00
Chris Lu
fc0cbf565f
add option to obfuscate the file names
2021-04-14 15:37:24 -07:00
Chris Lu
e75633c64f
volume.check.disk: break loop for read only volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/2002
2021-04-14 12:40:13 -07:00
Chris Lu
9d50867d08
volume.tier.move: avoid data loss when destination volume server already has the volume
...
fix https://github.com/chrislusf/seaweedfs/issues/2001
2021-04-14 10:26:26 -07:00
Chris Lu
ff4c1d5965
adjust logging
...
fix https://github.com/chrislusf/seaweedfs/issues/1999
2021-04-14 10:04:26 -07:00
qieqieplus
270645f8d7
fix #1996
2021-04-14 18:29:28 +08:00
Chris Lu
90677e1097
ensure to call line.Close()
...
fix https://github.com/chrislusf/seaweedfs/issues/1995
similar to https://github.com/peterh/liner/issues/104
2021-04-14 01:33:21 -07:00
Chris Lu
3f3268cd1b
go fmt
2021-04-14 00:30:16 -07:00
Chris Lu
ca0f07a188
move file reader, entryViewCache to file handle
...
reduce file object size
2021-04-14 00:29:58 -07:00
Chris Lu
5985a7d38d
add log level possible values
...
fix https://github.com/chrislusf/seaweedfs/issues/1989
2021-04-12 21:15:51 -07:00
Chris Lu
f5de42fae3
Merge pull request #1975 from kmlebedev/iam_handlers
...
IAM handlers
2021-04-12 12:07:45 -07:00
Chris Lu
1e033d45b8
simpler logic
...
related to https://github.com/chrislusf/seaweedfs/pull/1981
2021-04-12 12:04:53 -07:00
Chris Lu
52200a903b
Merge pull request #1981 from ueni-ltd/if-modified-since
...
Fix If-Modified-Since behavior
2021-04-12 12:02:54 -07:00
Chris Lu
519b0e1e49
filer: upload to a directory without "/" suffix
...
fix https://github.com/chrislusf/seaweedfs/issues/1988
2021-04-12 11:56:56 -07:00
Konstantin Lebedev
8e02e138ea
Merge branch 'upstreamMaster' into iam_handlers
2021-04-12 11:22:51 +05:00
Chris Lu
742ab1ec81
2.39
2021-04-11 19:47:11 -07:00
Chris Lu
0df5b53ad8
adjust help message
2021-04-11 00:26:28 -07:00
Chris Lu
f62c153274
go fmt
2021-04-10 23:48:18 -07:00
Chris Lu
af313dff58
add gateway for easier POST and DELETE blobs
2021-04-10 23:47:47 -07:00
Konstantin Lebedev
5021bea698
GetUserPolicy
2021-04-10 23:57:45 +05:00
Chris Lu
0b82edc0d2
filer: avoid stuck uploader
...
fix https://github.com/chrislusf/seaweedfs/issues/1980
reverting the file upload batch executor
2021-04-09 13:05:15 -07:00
Chris Lu
93f4146ffa
properly release the view cache
2021-04-09 12:36:39 -07:00
Merlin Gaillard
f952f979d1
filer: return 304 when If-Modified-Since == Last-Modified
2021-04-09 15:04:17 +02:00
Merlin Gaillard
4d4acc715e
s3api: handle 304 response code from filer
2021-04-09 12:13:19 +02:00
Chris Lu
6deb647a8f
mount: fix possible memory leak
...
if many files are read repeatedly, their metadata are accumulated in memory. This fix cleared the metadata after the file is read.
2021-04-08 19:47:31 -07:00
Konstantin Lebedev
ba175f81b5
add auth aws signV4
2021-04-08 17:40:47 +05:00
Konstantin Lebedev
995ae91007
add DeleteUserPolicy
2021-04-08 11:16:36 +05:00
Philippe Pepiot
42a761ee20
Fix typo in weed filer long help
2021-04-07 22:47:23 +02:00
Chris Lu
3be061994f
skip connection reset error
...
fix https://github.com/chrislusf/seaweedfs/issues/1971
this is because the connections are pooled but the volume server has reset the connection
2021-04-07 00:54:13 -07:00
Chris Lu
c5b08bac1b
remove mac specific mount options
2021-04-07 00:54:13 -07:00
Konstantin Lebedev
f5f8eec8e2
fix get filerGrpcAddress
2021-04-06 13:53:56 +05:00
Konstantin Lebedev
011e6e90ee
Merge branch 'upstreamMaster' into iamapipr
2021-04-06 13:50:33 +05:00
Konstantin Lebedev
ed79baa30f
add tests
2021-04-06 13:43:08 +05:00
Chris Lu
2327c0756b
fix to avoid loop
2021-04-05 23:24:26 -07:00
Chris Lu
a37eca78cd
2.38
2021-04-05 19:41:54 -07:00
Chris Lu
c131764c34
ensure error is sent back in channel
...
fix https://github.com/chrislusf/seaweedfs/issues/1966
avoid shared readErr variable
2021-04-05 19:40:12 -07:00
Chris Lu
0f64f5b9c8
mount: add readOnly option
...
fix https://github.com/chrislusf/seaweedfs/issues/1961
2021-04-04 21:40:58 -07:00
Chris Lu
6eee200c13
2.37
2021-04-04 18:45:48 -07:00
Chris Lu
fbb82a5c9c
skip limiting if limit is zero
2021-04-04 18:38:33 -07:00
Chris Lu
8251d1140e
refactor
2021-04-04 18:38:33 -07:00
Chris Lu
5dca98fe71
fix typo
2021-04-03 11:37:13 -07:00
Chris Lu
5e64f65632
ensure tables are created
...
fix https://github.com/chrislusf/seaweedfs/issues/1957
2021-04-02 12:02:26 -07:00
Chris Lu
319743d330
go fmt
2021-04-02 02:22:59 -07:00
Chris Lu
8e404a1433
go fmt
2021-04-02 02:22:26 -07:00
Chris Lu
bdf2ddddfd
revert to same implementation as before
...
This reverts commit 7e8edc3c4a
.
2021-04-02 02:21:38 -07:00
Chris Lu
67e019d54b
add missing changes
2021-04-02 01:11:44 -07:00
Chris Lu
7e8edc3c4a
refactoring
2021-04-02 01:10:24 -07:00
Chris Lu
cc0df36a9e
smaller file chunks are faster
2021-04-01 02:21:59 -07:00
Chris Lu
cefe66f159
dedicated upload processor
...
avoid thundering effect of overloading volume servers
2021-04-01 02:21:40 -07:00
Chris Lu
1f984d2645
refactor buffer pool
2021-04-01 02:20:00 -07:00
Chris Lu
b5880334fc
refactor
2021-03-30 21:07:34 -07:00
Chris Lu
e79e2ddeed
remove unused MoveEvents
2021-03-30 20:42:44 -07:00
Chris Lu
e06807a55b
adjust logs
2021-03-30 20:36:06 -07:00
Chris Lu
ac875976c0
filer, volume: add concurrent upload size limit to avoid OOM
...
add some back pressure when writes are slow
2021-03-30 02:10:53 -07:00
Chris Lu
a1e18a1384
server add memory profiling
2021-03-30 02:10:53 -07:00
LazyDBA247-Anyvision
9385e2224d
fix postgres2 upsertQuery syntax also
2021-03-30 10:51:52 +03:00
Chris Lu
a5662c9e5e
adjust ordering
2021-03-30 00:26:57 -07:00
Chris Lu
bec74c3e6a
Merge pull request #1948 from LazyDBA247-Anyvision/master
...
add enableUpsert=true
2021-03-30 00:25:01 -07:00
LazyDBA247-Anyvision
9f1cab179c
use a compatible pg syntax for upsert
2021-03-30 10:21:27 +03:00
Chris Lu
a95929e53c
reduce default concurrentWriters to 32
2021-03-30 00:17:52 -07:00
LazyDBA247-Anyvision
7f44d953b5
fix GetBool
2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision
4c51e6a660
add enableUpsert=true
...
and rename config to upsertQuery
2021-03-30 00:32:03 +03:00
Chris Lu
b61e9f6a1a
adjust ordering
2021-03-29 00:49:50 -07:00
Chris Lu
1ebdbad222
Merge pull request #1946 from LazyDBA247-Anyvision/master
...
Adding Customisation - insertQuery for postgres/mysql
2021-03-29 00:42:48 -07:00
Konstantin Lebedev
8a95f9c10c
iam GetUser
2021-03-29 12:01:44 +05:00
LazyDBA247-Anyvision
96c62bd34d
scaffold - insertQuery
...
Example for upsert query for mysql & postgres variants
2021-03-29 10:01:12 +03:00
LazyDBA247-Anyvision
4a02389eb0
Adding custom insertQuery support for postgres/2 mysql/2
2021-03-29 09:58:13 +03:00
Chris Lu
6b7aa9633f
2.36
2021-03-28 19:09:06 -07:00
Chris Lu
bd7471d877
refactor
2021-03-25 12:05:59 -07:00
李海
06be5dc6c3
log snowflake sequencer nodeid's hex when start, it'll be in part of new assigned fid
2021-03-25 18:59:40 +08:00
李海
69b2dab9c6
add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id
2021-03-25 18:49:26 +08:00
Konstantin Lebedev
40938d6a47
SaveInsideFiler S3 Configuration
2021-03-25 12:34:11 +05:00
Konstantin Lebedev
9f26f2815c
SaveAs S3 Configuration
2021-03-25 12:34:11 +05:00
Konstantin Lebedev
82b0463fac
handler PutUserPolicy
...
https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html
2021-03-25 12:34:11 +05:00
Konstantin Lebedev
d7719d0542
base handlers
2021-03-25 12:34:11 +05:00
Konstantin Lebedev
03c7953254
init Iam Api Server
2021-03-25 12:34:11 +05:00
Chris Lu
82dfe06066
Merge pull request #1933 from kmlebedev/missingKeysThreshold
...
add missingKeysThreshold for fsck
2021-03-24 10:56:25 -07:00
Konstantin Lebedev
020a5d40c3
avoid counter is null
2021-03-24 22:51:22 +05:00
Konstantin Lebedev
df6cf0a2fa
nonRepairThreshold
2021-03-24 22:07:13 +05:00
Konstantin Lebedev
9ce82a9cd2
GOMAXPROCS use defaults to the value of runtime.NumCPU
...
https://github.com/chrislusf/seaweedfs/issues/1942
2021-03-24 18:45:58 +05:00
Konstantin Lebedev
60972f1c97
rename option repairThreshold
2021-03-24 13:24:49 +05:00
Chris Lu
a801332b0d
filer: return 409 if file conflicts with a directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1938
2021-03-24 01:03:11 -07:00
Chris Lu
c42b95c596
more help message
...
fix https://github.com/chrislusf/seaweedfs/issues/1937
2021-03-23 17:27:57 -07:00
Konstantin Lebedev
40a9e88c07
rename var
2021-03-23 21:42:35 +05:00
Konstantin Lebedev
d21a5bf5d0
add missingKeysThreshold for fsck
2021-03-23 15:04:07 +05:00
Chris Lu
85cff10787
mysql: ensure name column is case sensitive
...
fix https://github.com/chrislusf/seaweedfs/issues/1931
2021-03-23 00:46:50 -07:00
Chris Lu
4abb511db3
make a local copy of the in memory cached data
2021-03-22 22:33:07 -07:00
Chris Lu
3cbc40fa48
avoid creating multiple reader
2021-03-22 22:32:47 -07:00
Chris Lu
2f7c7afdec
refactor: remove unused parameter
2021-03-22 22:13:19 -07:00
Chris Lu
d48dd0c738
use []byte directly instead of bytes.Buffer
2021-03-22 22:12:57 -07:00
Chris Lu
8eb14967ef
shell: volume.check.disk adds optional slow mode
...
fix https://github.com/chrislusf/seaweedfs/issues/1930
2021-03-22 21:01:43 -07:00
Chris Lu
288369cfc7
mount: release resources when Forget() is called
...
address https://github.com/chrislusf/seaweedfs/issues/1929
2021-03-22 16:22:19 -07:00
Chris Lu
5d931eff27
avoid possible nil
...
fix https://github.com/chrislusf/seaweedfs/issues/1928
The nil was because of `dn.Parent().UnlinkChildNode(dn.Id())` in topo.UnRegisterDataNode() function, when the dn leaves the cluster.
2021-03-22 13:24:07 -07:00
Chris Lu
f315eb2bb8
fix build error
2021-03-22 00:12:53 -07:00
Chris Lu
1dd5bc134c
2.35
2021-03-22 00:05:09 -07:00
Chris Lu
b465095db1
shell: add volume.check.disk to fix inconsistency for replicated volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1923
2021-03-22 00:03:16 -07:00
Chris Lu
df461402cc
ensure entry attribute is not nill
...
fix https://github.com/chrislusf/seaweedfs/issues/1926
2021-03-21 22:27:30 -07:00
Chris Lu
102a951377
refactor, split into 2 files
2021-03-21 13:05:53 -07:00
Chris Lu
f6e2566ee5
better logs for cookie mismatch
2021-03-20 12:02:57 -07:00
Chris Lu
27e24162f2
minor
2021-03-20 06:37:03 -07:00
Chris Lu
41cc6e661b
fs.mv ensures there are 2 arguments
2021-03-20 06:34:13 -07:00
Chris Lu
b2c1c209a5
refactor
2021-03-20 06:34:13 -07:00
Chris Lu
e9b1853d63
go fmt
2021-03-19 01:31:56 -07:00
Chris Lu
b1a86cf808
s3: copy object to itself
...
fix https://github.com/chrislusf/seaweedfs/issues/1922
2021-03-19 01:31:49 -07:00
Konstantin Lebedev
bf94149920
add line
2021-03-17 23:41:34 +05:00
Konstantin Lebedev
c5705e7a35
dir + slash key does not exist
...
https://github.com/chrislusf/seaweedfs/issues/1917
2021-03-17 23:40:42 +05:00
Konstantin Lebedev
eb54993a4e
Merge branch 'upstreamMaster' into check_chunkviews_mr
...
# Conflicts:
# weed/filer/filechunk_manifest.go
# weed/filer/stream.go
# weed/replication/repl_util/replication_util.go
# weed/util/fasthttp_util.go
2021-03-16 15:29:49 +05:00
Chris Lu
9672f9e1b2
2.34
2021-03-16 03:01:15 -07:00
Chris Lu
69694a17be
reverting 7d57664c2d
2021-03-16 02:59:26 -07:00
Chris Lu
c00dd5e62e
report error if entry not found
2021-03-16 02:55:49 -07:00
Konstantin Lebedev
06da02739d
CheckAllChunkViews() for HEAD requests only
2021-03-16 14:15:17 +05:00
Chris Lu
91a3ac9731
2.33
2021-03-16 00:36:06 -07:00
Chris Lu
4b1ed227d1
revert fasthttp changes
...
related to https://github.com/chrislusf/seaweedfs/issues/1907
2021-03-16 00:33:14 -07:00
Chris Lu
10164d0386
Merge pull request #1905 from wuh-fnst/s3ListCounterErr
...
s3: count correction for the number of files in the dir
2021-03-15 20:54:58 -07:00
Chris Lu
22a07a12d2
[mount] [regression] filer.path
option results in empty mount in 2.32
...
fix https://github.com/chrislusf/seaweedfs/issues/1906
2021-03-15 20:46:25 -07:00
wuh-fnst
050a7f9599
s3: count correction for the number of files in the dir
2021-03-16 09:22:03 +08:00
Konstantin Lebedev
3a3699867b
Status PartialContent for Content-Range response
2021-03-15 23:30:22 +05:00
Konstantin Lebedev
90510e3137
If WriteHeader is not called explicitly, the first call to Write
...
will trigger an implicit WriteHeader(http.StatusOK).
WriteHeader: Only one header may be written. Go does not currently!
2021-03-15 21:22:59 +05:00
Konstantin Lebedev
46b9f5cff4
add debug logging
2021-03-15 19:33:33 +05:00
Konstantin Lebedev
7194a5e7bf
avoid http error: superfluous response.WriteHeader
...
https://github.com/chrislusf/seaweedfs/issues/1838
2021-03-15 18:52:59 +05:00
Chris Lu
9f00f95bfb
2.32
2021-03-14 21:32:00 -07:00
Chris Lu
a4cfffc264
shell: fix moving volume, volume server evacuate
...
fix https://github.com/chrislusf/seaweedfs/issues/1534
2021-03-14 21:29:55 -07:00
Chris Lu
ca100568f7
fix parsing
2021-03-14 20:50:14 -07:00
Chris Lu
352ba23f83
revert previous change
...
revert 29e62aba00
2021-03-14 20:49:56 -07:00
Chris Lu
29e62aba00
possible fix for volume balance
...
address https://github.com/chrislusf/seaweedfs/issues/1534
2021-03-14 20:23:19 -07:00
Chris Lu
35d939a1d2
go fmt
2021-03-14 13:21:02 -07:00
Chris Lu
2d4c2db81d
filer: leveldb, rocksdb auto create store directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1901
2021-03-14 13:20:14 -07:00
Chris Lu
6d3a96eb56
filer: mysql2, postgres2 trigger actions on bucket creation and deletion
...
fix https://github.com/chrislusf/seaweedfs/issues/1877
2021-03-13 22:07:39 -08:00
Chris Lu
cb423312a4
prevent nil volume message
2021-03-13 11:05:29 -08:00
Chris Lu
972327f966
prevent nil volume nm
2021-03-13 11:04:51 -08:00
Chris Lu
2a68ddb963
default to empty host to bind to tcp4 and tcp6
2021-03-12 15:32:51 -08:00
Chris Lu
5fbcaaf37a
fullpath() test is not needed
2021-03-12 09:45:25 -08:00
Chris Lu
7d57664c2d
mount: internals switch to filer.Entry instead of protobuf
2021-03-12 00:36:38 -08:00
wuh-fnst
4c1d945e46
make List correctly judge whether it is the last file
2021-03-12 11:22:26 +08:00
Chris Lu
cca66c7fbe
print out the url
2021-03-11 18:39:45 -08:00
Chris Lu
7f887c78d2
fix
2021-03-11 18:18:16 -08:00
Chris Lu
fcc52d1e4e
shell: add s3.clean.uploads to clean up stale multipart uploads
...
fix https://github.com/chrislusf/seaweedfs/issues/1855
2021-03-11 18:05:24 -08:00
Chris Lu
40dc5ac904
mount: add a bit retry when connecting to filer during startup
2021-03-11 16:36:43 -08:00
Chris Lu
aba47fd9e8
reduce repeated calls to dir.FullPath()
2021-03-11 14:08:20 -08:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder
...
s3 delete-objects pass OK for fail to delete non-empty folder
2021-03-11 11:01:44 -08:00
Chris Lu
0e35836cc1
Merge pull request #1890 from kmlebedev/ZeroDataSize
...
error if read chunk zero data size
2021-03-11 10:51:48 -08:00
Konstantin Lebedev
58cdcc6d6e
error if urls not found
2021-03-11 23:34:36 +05:00
Chris Lu
79280e1513
escape special characters
...
fix https://github.com/chrislusf/seaweedfs/issues/1884
2021-03-11 09:49:40 -08:00
Konstantin Lebedev
4db21012c1
error if read chunk zero data size
2021-03-11 22:38:59 +05:00
wuheng
828fbf3fb7
s3: "isLast" returns true when the file does not exist
2021-03-11 15:20:50 +08:00
Chris Lu
5d446673a6
fix error printing
2021-03-10 17:13:04 -08:00
Chris Lu
ca546f47e1
s3: escape object key if containing special characters
...
fix https://github.com/chrislusf/seaweedfs/issues/1884
2021-03-10 13:19:28 -08:00
Chris Lu
105fcd7bfc
Merge pull request #1872 from kmlebedev/grpc_auth
...
TLS allowed CommonNames
2021-03-10 11:17:49 -08:00
Konstantin Lebedev
bb3af2d70b
no directoriesWithDeletion
2021-03-10 19:52:41 +05:00
Konstantin Lebedev
15b018da34
s3 delete-objects pass OK for fail to delete non-empty folder
2021-03-10 19:41:35 +05:00
Konstantin Lebedev
348e21a08c
add comments
2021-03-10 14:42:39 +05:00
Konstantin Lebedev
831953c55c
allowed wildcard domain
2021-03-10 14:02:13 +05:00
Konstantin Lebedev
4bf93d6e63
comma-separated
2021-03-10 12:43:13 +05:00
Konstantin Lebedev
0e02f7e258
comma-separated SSL certificate common names
2021-03-10 12:42:44 +05:00
Chris Lu
737bde5ab7
refactoring, remove duplicated code
2021-03-09 23:23:01 -08:00
Chris Lu
5ba4b479f8
properly lock file.entry object
...
fix https://github.com/chrislusf/seaweedfs/issues/1882
2021-03-09 23:08:38 -08:00
Chris Lu
5be83dd4a9
add comments
2021-03-09 14:13:48 -08:00
Chris Lu
9d8ca513d9
refactor
2021-03-09 13:21:26 -08:00
Chris Lu
f5b5e4da2d
2.31
2021-03-09 12:52:16 -08:00
Chris Lu
828f6e9f4d
volume: auto add missing vif files
...
fix https://github.com/chrislusf/seaweedfs/issues/1878
2021-03-09 12:09:32 -08:00
Chris Lu
387c6f4218
Revert "delete the folder object in multi_object_delete"
...
This reverts commit 2e89c8c9ae
.
2021-03-09 10:07:27 -08:00
qieqieplus
5b16820924
fix: collection stats won't update if all volumes expired at same iteration
2021-03-09 18:49:45 +08:00
Konstantin Lebedev
190fada1ef
TLS allowed commonNames
2021-03-08 21:39:44 +05:00
Konstantin Lebedev
c6d3735605
permitCommonNames
...
https://github.com/chrislusf/seaweedfs/issues/1841
https://jbrandhorst.com/post/grpc-auth/
2021-03-08 13:16:17 +05:00
Chris Lu
d888e9a90f
2.30
2021-03-07 15:01:19 -08:00
Chris Lu
60686a0bb7
make tcp optional
2021-03-07 14:45:36 -08:00
Chris Lu
3b2a9c98ef
fix nil
2021-03-07 11:45:35 -08:00
Chris Lu
df72dc206d
stats collect chunk upload retry count
2021-03-07 11:26:15 -08:00
Chris Lu
726edab054
avoid nil when closing an index
...
fix https://github.com/chrislusf/seaweedfs/issues/1870
2021-03-07 11:03:09 -08:00
Chris Lu
bdfed16d42
avoid nil exception
...
fix https://github.com/chrislusf/seaweedfs/issues/1869
2021-03-07 01:49:06 -08:00
Chris Lu
1bd880dcdb
adds tcp writes benchmark
2021-03-06 14:26:27 -08:00
Chris Lu
9d402ebe9f
refactoring
2021-03-06 14:26:27 -08:00
Chris Lu
e9eddfb8a2
remove wrong writes
2021-03-06 14:26:27 -08:00
Chris Lu
8363be8548
trim out trailing return character
2021-03-06 14:26:27 -08:00
Chris Lu
38fc200e56
CRCWriter consistent with CRC
2021-03-06 14:26:27 -08:00
Chris Lu
1444e9d275
migrated multi host connection pool from godropbox package
...
removing unneeded dependencies, which involved etcd versions.
2021-03-06 14:26:27 -08:00
Patrick Schmidt
7413d59750
Fix EC shard count logic
...
This fixes the calculation of the amount of EC shards a node holds.
Previously a global counter was increased, but also used inside the
loop to apply disk usage deltas. This led to wrong absolute numbers.
The fix is to apply only deltas of single EC shards per iteration.
2021-03-05 12:50:58 +01:00
Chris Lu
400de380f4
volume server: support tcp direct put/get/delete
2021-03-05 02:29:38 -08:00
Chris Lu
2e89c8c9ae
delete the folder object in multi_object_delete
...
fix https://github.com/chrislusf/seaweedfs/issues/1857
2021-03-04 14:03:40 -08:00
Chris Lu
c5cb346a08
fix compilation error
2021-03-04 12:36:20 -08:00
Chris Lu
f5cc96f289
skip checking master peers if not starting master
2021-03-04 12:31:26 -08:00
Chris Lu
40ff30b83f
adjust help message
2021-03-03 02:12:46 -08:00
Chris Lu
c0842fe99f
add filer.meta.backup command
2021-03-03 02:02:29 -08:00
Chris Lu
0c5c51eb98
adjust help message
2021-03-02 23:07:39 -08:00
Chris Lu
bd727b7b53
go fmt
2021-03-02 20:59:56 -08:00
Chris Lu
e982b06bcd
support IPv6
2021-03-02 20:59:39 -08:00
Chris Lu
bcf32591b7
reduce possibility of nil entry
2021-03-02 13:33:56 -08:00
Chris Lu
be9c7c21ec
reduce possibility of nil file entry
2021-03-02 11:17:17 -08:00
Chris Lu
02e146f3e1
return err
2021-03-02 08:54:18 -08:00
Chris Lu
5511722420
s3: list permission is needed to list my buckets
...
fix https://github.com/chrislusf/seaweedfs/issues/1837
2021-03-01 12:41:55 -08:00
Chris Lu
c46d16b2f3
skip network error when remote volumes start
2021-03-01 01:20:06 -08:00
Chris Lu
6e43e8ce17
add logs for remote file loading
2021-03-01 00:48:59 -08:00
Chris Lu
015d16f43f
add vif file versions in case loading superblock fails
2021-03-01 00:48:30 -08:00
Chris Lu
d680676d45
skip already loaded backends
2021-03-01 00:47:03 -08:00
Chris Lu
540441fd38
go fmt
2021-02-28 20:34:14 -08:00
Chris Lu
c9722dceb2
adjust log level
2021-02-28 20:26:55 -08:00
Chris Lu
1b1c018165
adjust text
2021-02-28 19:02:43 -08:00
Chris Lu
2cca07b44c
cloud tier to non-AWS s3 gateways
2021-02-28 18:59:09 -08:00
Chris Lu
f29ae0db76
2.29
2021-02-28 18:08:26 -08:00
bingoohuang
5cdff56731
set default env prefix to WEED_
2021-03-01 09:35:23 +08:00
Chris Lu
e52c94640e
filer.backup: added to replace filer.replicate
2021-02-28 16:22:27 -08:00
Chris Lu
014a31d11a
minor
2021-02-28 16:21:09 -08:00
Chris Lu
f2fcb77808
local file sink: create backup files with permission 0755
2021-02-28 16:20:47 -08:00
Chris Lu
984fdd6192
always use non bucket prefixing url
2021-02-28 16:20:13 -08:00
Chris Lu
9abb041763
filer source: support filerProxy mode
2021-02-28 16:19:47 -08:00
Chris Lu
678c54d705
data sink: add incremental mode
2021-02-28 16:19:03 -08:00
Chris Lu
4ff2c5c4c9
rename file
2021-02-28 16:14:21 -08:00
Chris Lu
3b76a51f5f
add help message
2021-02-27 12:15:49 -08:00
Chris Lu
70434df105
use "options"
2021-02-27 12:12:53 -08:00
Chris Lu
d3c31c69a7
avoid confusion: conf and config are too similar
2021-02-27 12:08:09 -08:00
Chris Lu
543c8daa6d
Merge pull request #1844 from kmlebedev/processRangeRequest
2021-02-26 09:18:22 -08:00
Patrick Schmidt
5f7b024891
Show the real disk usage in stats calls
...
Currently the file size of only one volume location is taken into
account in the stats. This commit multiplies the disk usages by the
amount of nodes holding a replica of the volume.
This will yield the expected amount of disk usage and matches the
total size calculations from before.
2021-02-26 13:58:40 +01:00
Konstantin Lebedev
ee21c0042e
log error write entry
2021-02-26 14:18:01 +05:00
Chris Lu
7ba75e3d5a
filer: do not return no content for empty files
...
fix https://github.com/chrislusf/seaweedfs/issues/1831
fix https://github.com/chrislusf/seaweedfs/issues/1830
2021-02-23 12:25:27 -08:00
Chris Lu
9edd964627
volume.tier.move: avoid repeated move for replicated volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-784139348
2021-02-23 03:49:14 -08:00
Chris Lu
37f104f88f
2.28
2021-02-22 22:54:34 -08:00
Chris Lu
90cdf9dcac
avoid conflict with "weed scaffold -config=xxx"
2021-02-22 16:57:27 -08:00
Chris Lu
72b0d9d8c4
avoid unnecessary user home checking
2021-02-22 16:50:56 -08:00
Chris Lu
2270737344
volume: avoid fixed vacuum timeout for large volumes
...
1GB for 3 minutes, about 5.7MB/s
2021-02-22 12:52:37 -08:00
Chris Lu
44bdfb2d15
filer: avoid encryption and compression at the same time
...
fix https://github.com/chrislusf/seaweedfs/issues/1828
2021-02-22 12:22:49 -08:00
Chris Lu
62191b08ea
disk type support custom tags
2021-02-22 02:03:12 -08:00
Chris Lu
30b30b8fe0
volume.tier.move: passing non-empty disk type
2021-02-22 01:59:03 -08:00
Chris Lu
5da63e045e
avoid moving to another server with the same volume id
2021-02-22 01:44:18 -08:00
Chris Lu
6a4546d2c0
shell: add volume.tier.move
2021-02-22 01:30:07 -08:00
Chris Lu
1c233ad986
refactoring
2021-02-22 00:28:42 -08:00
Chris Lu
151c281f36
2.27
2021-02-21 19:29:27 -08:00
Chris Lu
258e93bc86
Merge pull request #1823 from bingoohuang/master
...
feature: support command line arguments in a configuration file
2021-02-20 22:35:18 -08:00
Chris Lu
03c643aa6b
fix test
...
fileSize need to be divided by 8
2021-02-20 21:19:21 -08:00
Chris Lu
7635f6b9fa
disk file avoid file.Stat()
2021-02-20 20:06:06 -08:00
bingoo
1af6c96cd0
fix log help
2021-02-21 10:46:01 +08:00
Chris Lu
a2383b3b12
Revert "Revert "avoid file.Stat()""
...
This reverts commit 7ef4c24f18
.
2021-02-20 12:42:09 -08:00
Chris Lu
7ef4c24f18
Revert "avoid file.Stat()"
...
This reverts commit 98c93ca465
.
2021-02-20 12:39:33 -08:00
Chris Lu
1ad3200094
skip seek() when index file writes
2021-02-20 12:39:25 -08:00
Chris Lu
98c93ca465
avoid file.Stat()
...
avoid one Syscall, but did not help on performance though
2021-02-20 09:44:17 -08:00
bingoohuang
ee082ae402
revert comment temporarily
2021-02-20 16:59:38 +08:00
bingoohuang
ee7cdf3668
revert ErrXyz to ErrorXyz temporarily.
2021-02-20 16:58:08 +08:00
bingoohuang
50df484d86
support command line arguments in a configuration file so we can weed master -config=master.conf
, weed volume -config=volume.conf
and etc.
2021-02-20 16:52:57 +08:00
bingoohuang
f69356f589
Merge pull request #3 from chrislusf/master
2021-02-20 16:45:02 +08:00
Chris Lu
565f7a6e72
Update data_node.go
2021-02-19 14:22:36 -08:00
Chris Lu
a37473ae60
add back volume ids
...
address https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-782339576
2021-02-19 14:22:12 -08:00
Chris Lu
a1210d1e8d
Revert "volume: avoid file.stat(), file.seek() if possible during writes"
...
This reverts commit c78409a598
.
2021-02-19 03:56:27 -08:00
Chris Lu
b961cd6208
add WIP message
2021-02-19 03:39:19 -08:00
Chris Lu
f465d63b5d
grpc should fail when heart beating to master
...
fix https://github.com/chrislusf/seaweedfs/issues/1820
2021-02-19 03:37:29 -08:00
Chris Lu
c576ad04ac
fix volume server display for volumes
2021-02-19 01:38:56 -08:00
bingoohuang
7ffe736d20
renaming to ErrXyz
2021-02-19 15:59:55 +08:00
Chris Lu
c78409a598
volume: avoid file.stat(), file.seek() if possible during writes
2021-02-18 23:22:54 -08:00
Chris Lu
29575dadc5
remove extra files
2021-02-18 19:14:58 -08:00
Chris Lu
73958e357d
add descriptive error if no free volumes
2021-02-18 19:10:20 -08:00
Chris Lu
776f497469
filer: fs.configure should try to read from entry.content also
...
related to https://github.com/chrislusf/seaweedfs/issues/1792
2021-02-18 17:07:02 -08:00
Chris Lu
e1992c83de
webdav adds disktype
2021-02-18 15:45:44 -08:00
Chris Lu
c2ad6f1047
webdav add replication setting
...
fix https://github.com/chrislusf/seaweedfs/issues/1817
2021-02-18 12:15:09 -08:00
Chris Lu
816ef0ed30
add more help message
...
fix https://github.com/chrislusf/seaweedfs/issues/1816
2021-02-18 10:34:49 -08:00
bingoohuang
352ac2f271
Merge remote-tracking branch 'origin/master'
2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34
use backticks instead of double quotes to avoid escaped additionally in regex
2021-02-18 14:05:28 +08:00
Chris Lu
3575d41009
go fmt
2021-02-17 20:57:08 -08:00
Chris Lu
6daa932f5c
refactoring to get master function, instead of passing master values directly
...
this will enable retrying later
2021-02-17 20:55:55 -08:00
Chris Lu
dd9f3a0104
add sleep between upload retries
2021-02-17 20:55:11 -08:00
Chris Lu
aa17311063
s3: set filename when uploading file chunks
2021-02-17 20:54:53 -08:00
Chris Lu
3f8b0da677
filer: do not print password on error
...
fix https://github.com/chrislusf/seaweedfs/issues/1809
2021-02-17 02:13:52 -08:00
Chris Lu
cd866664a8
skip JWT if fileId is empty
...
related to https://github.com/chrislusf/seaweedfs/issues/1808
2021-02-16 15:39:12 -08:00
Chris Lu
632c94d438
Merge branch 'extend_to_disk_type'
2021-02-16 11:06:13 -08:00
Chris Lu
0611233f16
adjust printing
2021-02-16 10:55:30 -08:00
Chris Lu
68775d29e3
fix tests
2021-02-16 10:51:03 -08:00
Chris Lu
7403cd43c5
only use "" for hdd to avoid two values for the same thing
2021-02-16 10:50:42 -08:00
Chris Lu
b314d78e97
fix print
2021-02-16 10:48:28 -08:00
Chris Lu
53ca7e66ef
avoid dead lock
2021-02-16 10:48:16 -08:00
Chris Lu
38efc6f572
simplify
2021-02-16 10:47:32 -08:00
Chris Lu
ad70107eb4
ec balance among nodes with HDD
2021-02-16 06:22:50 -08:00
Chris Lu
3097b9a9b7
fix existence checking
2021-02-16 05:59:43 -08:00
Chris Lu
ebf320ec22
avoid divided by zero
2021-02-16 05:59:24 -08:00
Chris Lu
a0c6db361c
avoid nil
2021-02-16 05:33:38 -08:00
Chris Lu
36f95e50a9
avoid possible nil disk info
2021-02-16 05:13:48 -08:00
Chris Lu
43101ccea0
move to the empty nodes first
2021-02-16 04:27:16 -08:00
Chris Lu
38bbef7ec1
avoid nil
2021-02-16 04:16:46 -08:00
Chris Lu
cb9cc29518
volume.list display; fix updating maxVolumeCount for disk
2021-02-16 03:55:24 -08:00
Chris Lu
3fe628f04e
use hdd instead of empty string
2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab
this can compile now!!!
2021-02-16 02:47:02 -08:00
Chris Lu
b9b5b932c5
filer: postgres2 mysql2 avoid repeatedly creating tables
2021-02-15 16:19:24 -08:00
Chris Lu
71f0c19515
2.26
2021-02-15 13:38:38 -08:00
Chris Lu
8b169aa47c
s3: Critical - S3 Delete Multiple Objects, delete the bucket
...
fix https://github.com/chrislusf/seaweedfs/issues/1806
2021-02-15 13:38:29 -08:00
Chris Lu
50202d1391
Merge pull request #1805 from LazyDBA247-Anyvision/master
...
better postgresql connection pool management
2021-02-14 22:26:29 -08:00
Chris Lu
8726ef34c1
2.25
2021-02-14 22:23:43 -08:00
LazyDBA247-Anyvision
7f458d5e78
better postgres connection pool management
...
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime )
to enable refresh of stale connections.
2021-02-15 07:45:09 +02:00
Chris Lu
0bc3a1f9e8
disk type only supports hdd and ssd, not ready for random tags yet
2021-02-14 11:38:43 -08:00
LazyDBA247-Anyvision
51b4963e2e
postgres2 & memsql2
...
add escape (quote identifiers) for the dynamic sql
so tables (collections) with special characters will work.
2021-02-14 13:14:36 +02:00
Chris Lu
876dbe3d26
collect disk types
2021-02-13 23:25:16 -08:00
Chris Lu
861f8fab9b
refactoring
2021-02-13 22:34:12 -08:00
Chris Lu
881f05f5a0
adjust help message
2021-02-13 16:13:52 -08:00
Chris Lu
ef76365ec2
adjust help message
2021-02-13 15:47:08 -08:00
Chris Lu
7ce647f27e
support customizable disk type
2021-02-13 15:42:42 -08:00
Chris Lu
4bd8a692d8
disk type can be generic tags
2021-02-13 13:50:14 -08:00
Chris Lu
712b3e9e53
mount: release need to avoid nil reader
...
fix https://github.com/chrislusf/seaweedfs/issues/1803
2021-02-13 13:39:37 -08:00
Chris Lu
4ce56bac08
fix format error
2021-02-12 06:44:26 -08:00
Chris Lu
0604afcda7
mount: remove unnecessary flush on release
2021-02-12 06:01:04 -08:00
Chris Lu
7d9dc3c6a2
use fasthttp lib to read
2021-02-12 05:59:52 -08:00
Chris Lu
487e435679
adjust http max idle connections per host
...
related to https://github.com/chrislusf/seaweedfs/issues/1802
2021-02-12 03:47:15 -08:00
Chris Lu
0f426ce34d
Merge pull request #1801 from kmlebedev/recoveringRabbitMQ
...
Do reconnect to RabbitMQ
2021-02-11 01:05:03 -08:00
Konstantin Lebedev
94eac4f00e
Do reconnect to RabbitMQ
...
https://github.com/google/go-cloud/issues/2958
https://github.com/chrislusf/seaweedfs/issues/1773
2021-02-11 13:59:36 +05:00
Chris Lu
885ca34748
volume: fail fast if idx files are missing
...
fix https://github.com/chrislusf/seaweedfs/issues/1796
2021-02-11 00:44:40 -08:00
Chris Lu
a0e84c4fbc
go fmt
2021-02-10 23:41:05 -08:00
Chris Lu
c959aaa19b
mount: load directory based on correct filer path
...
fix https://github.com/chrislusf/seaweedfs/issues/1799
2021-02-10 23:14:18 -08:00
Konstantin Lebedev
0cfed8c3cb
restart replication if RabbitMQ connection closed
...
https://github.com/google/go-cloud/issues/2958
2021-02-10 15:29:05 +05:00
Chris Lu
770393a48c
volume: add capability to change disk type when moving a volume
2021-02-09 23:58:08 -08:00
Chris Lu
821c46edf1
Merge branch 'master' into support_ssd_volume
2021-02-09 11:37:07 -08:00
Chris Lu
a833021132
fix refactoring left over
2021-02-07 23:03:03 -08:00
Chris Lu
5e4b5109dd
2.24
2021-02-07 22:29:43 -08:00
Chris Lu
15c60cbb26
close the grpc connection after 10 hours
...
related to https://github.com/chrislusf/seaweedfs/issues/1782
2021-02-07 03:50:01 -08:00
bingoohuang
7256902fb0
fix typo offset.ToAcutalOffset to offset.ToActualOffset
2021-02-07 12:11:51 +08:00
bingoohuang
94ea3bd3a5
renaming NeedleMapType to NeedleMapKind
2021-02-07 09:00:03 +08:00
Chris Lu
c3af72d950
fier store: fix elastic search regression
...
fix https://github.com/chrislusf/seaweedfs/issues/1774
2021-02-04 01:30:14 -08:00
Konstantin Lebedev
2fb4c60540
fix log warn
2021-02-03 21:34:32 +05:00
Konstantin Lebedev
24ec17219b
RabbitMQ delay retry with Dead Letter Exchange
...
https://github.com/chrislusf/seaweedfs/issues/1773
https://github.com/google/go-cloud/issues/2952
2021-02-03 21:32:30 +05:00
Chris Lu
a331bbb3ae
filer: should return 204 on DELETE to nonexistent file
...
related to
https://github.com/chrislusf/seaweedfs/issues/1776
https://github.com/chrislusf/seaweedfs/issues/1160
2021-02-03 00:40:31 -08:00
Chris Lu
5353e38469
S3: should return 204 on DELETE to nonexistent file
...
fix https://github.com/chrislusf/seaweedfs/issues/1776
2021-02-03 00:35:44 -08:00
Chris Lu
2396ac234c
filer: fix elastic search pagination
...
possible fix for https://github.com/chrislusf/seaweedfs/issues/1774
2021-02-02 11:34:12 -08:00
Chris Lu
609daaf387
s3: DeleteMultipleObjectsHandler clean up leftover empty folders
...
fix https://github.com/chrislusf/seaweedfs/issues/1772
2021-02-01 10:49:17 -08:00
Chris Lu
d67ccb66c0
webdav: can start together with "weed server" or "weed filer"
2021-01-31 22:16:52 -08:00
Chris Lu
76430790b9
webdav: cache to version specific folder
2021-01-31 22:02:03 -08:00
Chris Lu
318a3d2efc
2.23
2021-01-31 20:18:06 -08:00
Chris Lu
4be51c0701
filer: leveldb and hbase may miss files when listing large directories more than 1024
...
fix https://github.com/chrislusf/seaweedfs/issues/1768
2021-01-31 20:11:44 -08:00
Chris Lu
1102ae32c4
fix concurrent map reads
2021-01-31 18:26:26 -08:00
Chris Lu
314dc1c957
filer: etcd store fix listing
...
fix https://github.com/chrislusf/seaweedfs/issues/1767
2021-01-30 13:38:44 -08:00
Chris Lu
d475c89fcc
go fmt
2021-01-28 15:23:46 -08:00
Chris Lu
19295600f9
mount: change option name to volumeServerAccess, with publicUrl and filerProxy modes
2021-01-28 15:23:16 -08:00
Chris Lu
990fa69bfe
add back AdjustedUrl() related code
2021-01-28 14:36:29 -08:00
Chris Lu
9292796ec2
fmt
2021-01-28 14:29:14 -08:00
Chris Lu
c2bf1a88ac
delete from the deepest directory first when checking empty folders
2021-01-28 14:28:40 -08:00
Chris Lu
e9d8201925
s3: batch purge empty folders
2021-01-28 13:20:06 -08:00
Chris Lu
cf252fc0cd
mount: report error when Flush()
...
The error in Release() is not working.
See https://github.com/jaderhs/libfuse/blob/master/FAQ
related to https://github.com/chrislusf/seaweedfs/issues/1765
2021-01-28 04:46:37 -08:00
Chris Lu
9a06c35da4
replicate: incremental sink only contains new and updated files
...
address da08402ba2
2021-01-28 02:39:22 -08:00
Chris Lu
da08402ba2
replicate: use creation time for local incremental file sink
...
related to https://github.com/chrislusf/seaweedfs/pull/1762
2021-01-28 02:17:41 -08:00
Chris Lu
822f1ade9d
Merge pull request #1762 from kmlebedev/backupsink
...
replication to create time date directory
2021-01-28 02:03:33 -08:00
Konstantin Lebedev
be1062b7fc
rename in scaffold
2021-01-28 14:59:20 +05:00
Konstantin Lebedev
02fdc0a333
rename backup to local_incremental and use mtime
2021-01-28 14:56:13 +05:00
Chris Lu
5f72c388bc
Merge pull request #1764 from danlsgiga/handle-umask
...
Use provided umask on mount
2021-01-28 01:09:20 -08:00
Daniel Santos
d38613c49a
match upstream
2021-01-27 20:56:41 -07:00
Chris Lu
b81956bcb5
mount: invalidate kernel cache when mounted to a filer path
...
fix https://github.com/chrislusf/seaweedfs/issues/1752#issuecomment-768178422
2021-01-27 10:28:37 -08:00
Konstantin Lebedev
6b54ff9912
replication to create time date directory
2021-01-27 15:01:33 +05:00
Chris Lu
f20ec82a28
update help message
...
see https://github.com/chrislusf/seaweedfs/issues/1758#issuecomment-768123224
2021-01-27 00:48:31 -08:00
Chris Lu
1045ab8b07
avoid error message
2021-01-27 00:21:59 -08:00
Chris Lu
0488c84bea
adjust help message
2021-01-27 00:17:52 -08:00
Konstantin Lebedev
3634811408
replication to todays date directory
2021-01-27 12:45:58 +05:00
Chris Lu
5e07afb0f0
Merge pull request #1759 from kmlebedev/sink.local
...
replication to local disk storage
2021-01-26 22:31:54 -08:00
Chris Lu
b3f66199db
shell: volume.fix.replication adds collectionPattern with wildcard characters
...
fix https://github.com/chrislusf/seaweedfs/issues/1758
2021-01-26 22:30:39 -08:00
Daniel Santos
0c64d60fdb
Fix log message with correct mode
2021-01-26 21:45:52 -07:00
Daniel Santos
0aaaa9b470
Fix var assignment
2021-01-26 21:06:08 -07:00
Daniel Santos
5a761bb7a4
Enforce umask correctly
2021-01-26 20:42:10 -07:00
Chris Lu
ad2a20c8a5
notification add ack and nack
2021-01-26 11:08:44 -08:00
Konstantin Lebedev
612b7975a1
replication to local disk storage
2021-01-26 22:50:25 +05:00
Chris Lu
3a1d3d3413
mount: properly invalidate kernel node cache entry
...
fix https://github.com/chrislusf/seaweedfs/issues/1752
2021-01-26 02:50:53 -08:00
Chris Lu
cc1f3907ff
filer.sync: default filerProxy to true
2021-01-25 10:14:42 -08:00
Chris Lu
62ebb917e0
mount: avoid the need to specify different cache directory for different mount
...
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/11
2021-01-25 09:10:20 -08:00
Chris Lu
17f7c1c43f
Merge branch 'mount_from_outside_cluster'
2021-01-24 19:03:02 -08:00
Chris Lu
00707ec00f
mount: outsideContainerClusterMode proxy through filer
...
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
2021-01-24 19:01:58 -08:00
Chris Lu
9bcb28a3ea
avoid possible nil reader
...
fix https://github.com/chrislusf/seaweedfs/issues/1754
2021-01-24 17:19:35 -08:00
Chris Lu
2c5eac5705
2.22
2021-01-24 14:06:09 -08:00
Chris Lu
6ca10725b8
Revert "mount: when outside cluster network, use filer as proxy to access volume servers"
...
This reverts commit 096e088d7b
.
2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b
mount: when outside cluster network, use filer as proxy to access volume servers
2021-01-24 01:41:38 -08:00
Chris Lu
ccbdb38c89
s3: fix for listing objects if more than 1000 in the folder
...
many existing tools have max key set to 1000
2021-01-24 00:27:52 -08:00
Chris Lu
80b8692688
filer.sync: replicate outside of either cluster, only need to see filers
2021-01-24 00:01:44 -08:00
Chris Lu
f8dbb03bdd
filer: append operation returns final file size
2021-01-23 04:19:49 -08:00
Chris Lu
759482e2ac
filer: proxy all http methods to volume servers
2021-01-23 03:52:38 -08:00
Chris Lu
937cfacc01
filer: add "proxyToFileId" to reverse proxy to a volume server
2021-01-23 03:43:48 -08:00
Chris Lu
20ef3bb8d4
mount: use direct_io to avoid OS page cache
...
fix https://github.com/chrislusf/seaweedfs/issues/1752
2021-01-22 22:39:46 -08:00
Chris Lu
711c3f3939
filer: fix http status 206 setting
...
fix https://github.com/chrislusf/seaweedfs/issues/1753
2021-01-22 00:26:30 -08:00
Chris Lu
024c186a48
filer.meta.tail: output json format for easier parsing
2021-01-21 22:39:25 -08:00
Chris Lu
84f05787f8
mount: limit background requests and congestion threshold
2021-01-20 23:30:14 -08:00
Chris Lu
f98817cfe6
filer: support appending to a file
2021-01-20 13:40:32 -08:00
Chris Lu
1b8e3da295
refactoring
2021-01-20 12:59:54 -08:00
Chris Lu
1a3f859c58
filer store: postgres configuration adjust text
2021-01-19 18:12:07 -08:00
Chris Lu
b434f7e4e0
filer store: postgres adjust default config, add optioanl schema
2021-01-19 18:10:36 -08:00
Chris Lu
d5add83e85
filer store: add postgres2
2021-01-19 18:07:29 -08:00
Chris Lu
52a8f1470e
filer store: add mysql2
2021-01-19 17:21:50 -08:00
Chris Lu
93b3adba98
fix bucket creation
2021-01-19 15:55:51 -08:00
Chris Lu
fa0c8d5283
fix error
2021-01-19 14:05:48 -08:00
Chris Lu
4c5b752b04
restructuring sql stores
2021-01-19 13:53:16 -08:00
Chris Lu
96354208c5
scaffold: add YugabyteDB
2021-01-19 12:35:14 -08:00
Chris Lu
ca8f793978
refactor: prepare for bucket specific tables
2021-01-19 12:34:58 -08:00
Chris Lu
90ce1eec19
add more input params
2021-01-19 11:32:55 -08:00
Chris Lu
f13c082583
add memsql to comments
2021-01-19 11:31:55 -08:00
Chris Lu
e439b65e38
fix test
2021-01-18 01:48:00 -08:00
Chris Lu
c7197470fd
2.21
2021-01-18 01:25:04 -08:00
Chris Lu
5a3386e39b
Revert "Revert "mount: fake support for socket/block/character/fifo devices""
...
This reverts commit 3d5cb7eb86
.
2021-01-18 01:15:07 -08:00
Chris Lu
67faa56920
Revert "Revert "mount: add more entry type""
...
This reverts commit 4085b79d38
.
2021-01-18 01:14:58 -08:00
Chris Lu
7119e7b491
mount: ensure reading from dirty pages
2021-01-18 01:14:42 -08:00
Chris Lu
de876c795d
minor fix
2021-01-18 01:14:27 -08:00
Chris Lu
4085b79d38
Revert "mount: add more entry type"
...
This reverts commit d7d907be27
.
2021-01-18 00:19:19 -08:00
Chris Lu
3d5cb7eb86
Revert "mount: fake support for socket/block/character/fifo devices"
...
This reverts commit 61ef2d8658
.
2021-01-18 00:18:57 -08:00
Chris Lu
61ef2d8658
mount: fake support for socket/block/character/fifo devices
...
to pass pjdfstest. A distributed file system can not really support these things anyway.
2021-01-17 23:51:10 -08:00
Chris Lu
d7d907be27
mount: add more entry type
2021-01-17 23:46:39 -08:00
Chris Lu
389426bbb7
s3: listing may repeat on the edge
2021-01-17 22:59:31 -08:00
Chris Lu
09f49d1c04
refactoring
2021-01-16 19:52:15 -08:00
Chris Lu
a4063a5437
add stream list directory entries
2021-01-15 23:56:24 -08:00
Chris Lu
01dc8a43ba
fix rocksdb
2021-01-15 18:31:29 -08:00
Chris Lu
16ad74f477
go fmt
2021-01-14 23:11:27 -08:00
Chris Lu
f002e668de
change limit to int64 in case of overflow
2021-01-14 23:10:37 -08:00
Chris Lu
19e52fd414
pass along hasMore
2021-01-14 22:49:29 -08:00
Chris Lu
c0bcf6f6e1
implement sql changes
2021-01-14 22:44:22 -08:00
Chris Lu
5d4568b91f
implement elastic changes
2021-01-14 22:42:25 -08:00
Chris Lu
893cbc8482
implement c* changes
2021-01-14 22:38:34 -08:00
Chris Lu
5ef43b9b09
implement etcd changes
2021-01-14 22:35:56 -08:00
Chris Lu
54527f0326
implement hbase changes
2021-01-14 22:34:35 -08:00
Chris Lu
b5ceffe188
implement leveldb changes
2021-01-14 22:33:05 -08:00
Chris Lu
c64bfb0e2e
implement mongodb changes
2021-01-14 22:28:51 -08:00
Chris Lu
34a846009d
implement rocksdb changes
2021-01-14 22:24:26 -08:00
Chris Lu
d2ece29fff
implement redis changes
2021-01-14 22:21:31 -08:00
Chris Lu
9a50dbcda0
chagned api
2021-01-14 22:18:38 -08:00
Chris Lu
e1c7bc66f9
change interface
2021-01-14 22:17:49 -08:00
Chris Lu
4aaa97c789
skip not found error in case of race condition
2021-01-14 19:56:14 -08:00
Chris Lu
d0826c2f62
mount: avoid showing "topics" folder
...
fix https://github.com/chrislusf/seaweedfs/issues/1325
2021-01-14 19:03:15 -08:00
Chris Lu
698f58f7c4
filer, s3: add http status 206 as late as possible
...
fix https://github.com/chrislusf/seaweedfs/issues/1746
2021-01-14 02:59:40 -08:00
Chris Lu
fd1d8a2a09
fix compilation
2021-01-14 00:06:15 -08:00
Chris Lu
ae812c07d6
Merge pull request #1743 from LazyDBA247-Anyvision/master
...
MYSQL - Better Connection Pool Control
2021-01-14 00:04:44 -08:00
Chris Lu
f20743bb38
remove WIP status
2021-01-13 22:21:34 -08:00
LazyDBA247-Anyvision
8eed763b97
better mysql connection pool management
...
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime )
to enable refresh of connections.
2021-01-14 08:14:21 +02:00
Chris Lu
f17aa1d06c
skip directory loop if dropping the bucket
2021-01-13 13:49:04 -08:00
Chris Lu
ca73013453
leveldb3: ensure repeatable adding removing buckets
2021-01-13 13:48:48 -08:00
Chris Lu
e2c7e3fe6d
better locks
2021-01-13 13:20:33 -08:00
Chris Lu
819a85f59a
avoid viper concurrent access
2021-01-13 13:14:52 -08:00
Chris Lu
3fb2ed9093
filer.meta.tail: optionally submit metadata changes to ElasticSearch
2021-01-13 03:31:37 -08:00
Chris Lu
c5df2577f5
rename
2021-01-13 00:31:19 -08:00
Chris Lu
90c5077610
rename "weed watch" to "weed filer.meta.tail"
2021-01-12 18:48:01 -08:00
Chris Lu
5cd9c9e642
remove println
2021-01-12 18:47:38 -08:00
Chris Lu
def6d8e565
fix test
2021-01-12 12:32:54 -08:00
Chris Lu
1c7faf1100
Revert "close opened file"
...
This reverts commit 2880160113
.
2021-01-12 11:07:14 -08:00
Chris Lu
2880160113
close opened file
2021-01-12 11:03:08 -08:00
Chris Lu
0ef43a23a7
go fmt
2021-01-12 02:30:12 -08:00
Chris Lu
629c9962e7
filer: add leveldb3
...
support dedicated leveldb instance for each bucket
2021-01-12 02:29:44 -08:00
Chris Lu
361043e6c1
filer store: leveldb2 fix nil entry error if not found
2021-01-12 02:28:57 -08:00
Chris Lu
cfb9342a15
avoid concurrent map updates to viper
2021-01-12 02:28:13 -08:00
Chris Lu
38d516251e
Revert "mount: avoid fatal error"
...
This reverts commit a4f0bac9b7
.
2021-01-11 13:50:15 -08:00
Chris Lu
a4f0bac9b7
mount: avoid fatal error
2021-01-11 13:26:59 -08:00
Chris Lu
8a7302ce37
s3: KeyCount is zero
...
fix https://github.com/chrislusf/seaweedfs/issues/1741
2021-01-11 03:50:45 -08:00
Chris Lu
2c7148cd55
redis cluster: do not enable route by latency by default
2021-01-11 02:43:53 -08:00
Chris Lu
acc0d5cd2c
Revert "Revert "filer store: redis module upgrade""
...
This reverts commit dd0171e35c
.
2021-01-11 02:30:19 -08:00
Chris Lu
dd0171e35c
Revert "filer store: redis module upgrade"
...
This reverts commit 40161c9cec
.
2021-01-11 02:09:18 -08:00
Chris Lu
394513f598
filer: ensure seamless meta data updates
2021-01-11 02:08:55 -08:00
Chris Lu
6e12a3a490
refactoring
2021-01-11 02:08:26 -08:00
Chris Lu
62ee484d12
refactoring
2021-01-11 00:03:13 -08:00
Chris Lu
1efb51ba84
filer: change to saveToFilerLimit from cacheToFilerLimit
...
short circuit saving small files to volume server
2021-01-10 23:14:46 -08:00
Chris Lu
f0d3b3bf93
2.20
2021-01-09 22:55:22 -08:00
Chris Lu
d3d3f2fb9b
mount: default to 128 concurrent writers
2021-01-09 22:53:37 -08:00
Chris Lu
90b117acf1
update ccache version
2021-01-08 02:17:43 -08:00
Chris Lu
e2e0e30a51
filer: redis default to false for useReadOnly and routeByLatency
2021-01-08 01:12:44 -08:00
Chris Lu
40161c9cec
filer store: redis module upgrade
2021-01-08 01:08:44 -08:00
Chris Lu
30efaa66c6
server: add option to disable master
2021-01-07 02:35:47 -08:00
Chris Lu
07f4703bfc
Merge pull request #1738 from qieqieplus/rocksdb
...
fix rocksdb crash when list directory
2021-01-07 02:26:41 -08:00
qieqieplus
0f18592315
alter style
2021-01-07 18:21:48 +08:00
qieqieplus
c7e7431fd3
fix seek lastkey may reach EOF
2021-01-07 17:07:56 +08:00
Chris Lu
e327385644
go fmt
2021-01-06 04:22:00 -08:00
Chris Lu
2b76854641
add "weed filer.cat" to read files directly from volume servers
2021-01-06 04:22:00 -08:00
Chris Lu
95ecf0c72f
Merge pull request #1735 from qieqieplus/rocksdb
...
ignore decode error for non-entry data
2021-01-06 02:22:18 -08:00
qieqieplus
0764fccde7
ignore decode error for non-entry data
2021-01-06 17:42:43 +08:00
Chris Lu
ab57767d4b
Merge pull request #1730 from qieqieplus/rocksdb
...
impl: TTL per entry for rocksdb; fix package name
2021-01-05 13:24:45 -08:00
qieqieplus
b2e50f602f
fix time comparison
2021-01-05 19:36:40 +08:00
qieqieplus
bbae13e415
impl: TTL per entry for rocksdb; fix package name
2021-01-05 17:58:40 +08:00
henry
b42c5b71c1
Seaweed custom header are not visible to Vue or javascript
2021-01-05 13:01:29 +08:00
henry
1309dfe612
Seaweed custom header are not visible to Vue or javascript
2021-01-05 12:58:46 +08:00
henry
942a30f04c
remove sort
2021-01-05 12:24:32 +08:00
henry
a3c432dadc
Merge branch 'master' of https://github.com/fuyouyshengwu/seaweedfs
2021-01-05 11:17:29 +08:00
qieqieplus
ebb223c190
fix for rocksdb kv
2021-01-04 21:21:46 +08:00
chenqieqie
40b3207f18
fix #1726
2021-01-04 18:48:55 +08:00
henry
97a94eddab
Merge branch 'master' of https://github.com/fuyouyshengwu/seaweedfs
2021-01-04 14:50:01 +08:00
Chris Lu
2ce86f308e
2.19
2021-01-03 17:24:02 -08:00
Chris Lu
9c9ba3c209
nil related
...
related to https://github.com/chrislusf/seaweedfs/issues/1676
2021-01-03 12:25:58 -08:00
Chris Lu
4b5adc60b4
upload result changes to the right output variable
2021-01-03 01:44:22 -08:00
Chris Lu
039ad101b1
manifest file also need to detect compression
...
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753585046
2021-01-03 01:42:21 -08:00
Chris Lu
5f538c5f9d
Merge branch 'add_rocksdb'
2021-01-03 00:48:56 -08:00
Chris Lu
bdd166c031
adjust readme
2021-01-03 00:47:58 -08:00
Chris Lu
a04d8927a8
add scaffold
2021-01-03 00:44:52 -08:00
Chris Lu
4250a3a675
upload: add read default collection from master
...
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753576071
2021-01-02 23:43:01 -08:00
Chris Lu
f4d87f8b9d
works
...
performance seems about the same as leveldb for insert entries
2021-01-02 23:32:58 -08:00
Chris Lu
2c3c2c27d7
separate prefix from namePattern
...
fix https://github.com/chrislusf/seaweedfs/issues/1722
2021-01-01 20:23:23 -08:00
Peter Cai
cd2a9865ed
filer: respect Content-Type on PUT
...
This is needed for the S3 gateway to conform to the S3 protocol -- the
server should always respect the Content-Type set by client.
2020-12-31 11:32:45 +08:00
Chris Lu
3433accb1b
weed upload: add TTL
2020-12-30 15:46:07 -08:00
henry
14ddd15508
1 Distinguish between folders and common files, folders are listed first
...
2 Add some custom headers that need to be exposed
2020-12-31 00:03:22 +08:00
Chris Lu
adef4ddc87
mount: load security keys
...
fix https://github.com/chrislusf/seaweedfs/issues/1714
2020-12-29 11:51:38 -08:00
Chris Lu
89977123b6
shell: add shell.toml support
...
fix https://github.com/chrislusf/seaweedfs/issues/1664
2020-12-28 15:07:18 -08:00
Chris Lu
d2ead72918
2.18
2020-12-28 12:37:10 -08:00
Chris Lu
2e6cf541e7
filer: avoid deletion error when item is not found
2020-12-28 00:18:37 -08:00
Chris Lu
6d2c979999
s3: fix regression
...
fix https://github.com/chrislusf/seaweedfs/issues/1707
2020-12-27 21:09:45 -08:00
Chris Lu
90df52d001
s3: add command option "allowEmptyFolder" to filer and server
2020-12-27 21:09:06 -08:00
Chris Lu
da7e5aaa65
filer: optimize for less number of directory lookup
...
bottom up directory lookup
2020-12-27 20:46:17 -08:00
henry
7458ff9523
Solve the Chinese name disorder of download file
2020-12-28 11:28:33 +08:00
Chris Lu
561a2ff0bc
s3: add option for "alllowEmptyFolder"
2020-12-27 10:50:27 -08:00
Chris Lu
aa020ee7e7
s3: restore V2 auth type
...
The added headers were included in calculating the signatures, failing the comparison
2020-12-26 22:01:16 -08:00
Chris Lu
da87f6b265
remove unused code
2020-12-26 15:21:12 -08:00
Chris Lu
0a7c5f85a9
filer: add namePattern to search in current folder
2020-12-26 15:05:31 -08:00
Chris Lu
515b7632d7
2.17
2020-12-25 23:58:22 -08:00
Chris Lu
0a067944cc
filer: add retries during volume moving
...
fix https://github.com/chrislusf/seaweedfs/issues/1704
2020-12-25 02:32:55 -08:00
Chris Lu
090f85be4b
s3: support config action Admin:bucket
2020-12-25 00:38:56 -08:00
Chris Lu
75613b2cc7
Merge branch 'add_hbase'
2020-12-24 12:13:05 -08:00
Chris Lu
0e016bc7bd
hbase add ttl
2020-12-24 12:10:35 -08:00
Chris Lu
8e48a235e2
s3: avoid looping if the directory is empty
...
fix https://github.com/chrislusf/seaweedfs/issues/1701
2020-12-24 11:34:52 -08:00
Chris Lu
1620de08ae
added more logs
2020-12-24 01:52:06 -08:00
Chris Lu
a09cd28986
add debug messages
2020-12-24 01:42:15 -08:00
Chris Lu
75c6edba9e
filer: hbase add enabled flag
2020-12-24 00:19:16 -08:00
Chris Lu
9bf6c10505
fix prefix
2020-12-24 00:18:59 -08:00
Chris Lu
2fea8cfa0f
fix compilation
2020-12-23 23:54:15 -08:00
Chris Lu
53bc1ea25b
fix compilation
2020-12-23 23:53:46 -08:00
Chris Lu
b5e2be635a
adjust for directory listing
2020-12-23 23:49:22 -08:00
Chris Lu
c4a202ec41
fix wrong column family
2020-12-23 23:23:05 -08:00
Chris Lu
c3d1b3b5aa
hook up
2020-12-23 21:49:01 -08:00
Chris Lu
94e3757c08
fix
2020-12-23 21:45:16 -08:00
Chris Lu
64c48c9724
seems compiling
2020-12-23 21:44:53 -08:00
Chris Lu
15da5834e1
Merge branch 'master' into support_ssd_volume
2020-12-23 14:37:37 -08:00
Chris Lu
3be3635799
just log error when checking empty dir failed
2020-12-23 14:36:48 -08:00
Chris Lu
0ca9d89589
s3: break loop if error
...
fix #1701
2020-12-23 14:34:59 -08:00
Chris Lu
132f275d04
s3: log errors instead of stopping when checking isDirectoryAllEmpty has error
2020-12-23 12:21:21 -08:00
Chris Lu
db62090f88
s3: break loop if error
...
fix https://github.com/chrislusf/seaweedfs/issues/1701
2020-12-23 12:14:51 -08:00
Baptiste Mille-Mathias
385a4dec30
[shell] Clarify bucket replication setting order
2020-12-23 09:35:24 +01:00
Chris Lu
1c7e1295dc
Merge branch 'master' into support_ssd_volume
2020-12-22 17:44:52 -08:00
Chris Lu
da134a2eb7
minor
2020-12-22 17:43:13 -08:00
Chris Lu
beb3b8ddb0
shell: change bucket.list to s3.bucket.list, same for create and delete
2020-12-22 17:40:55 -08:00
Chris Lu
4cf3176cbb
filer first, master second
2020-12-22 17:34:44 -08:00
Chris Lu
a1009e8044
filer: add -defaultStoreDir so that filer.toml can be skipped
...
fix https://github.com/chrislusf/seaweedfs/issues/1659
2020-12-22 17:33:40 -08:00
Chris Lu
a0990b929d
minor
2020-12-22 17:33:40 -08:00
Baptiste Mille-Mathias
0ed5345954
[shell] Add more information to bucket.create
...
State the replication setting that will be honor if not defined.
2020-12-22 17:18:59 +01:00
Chris Lu
97e3432dfe
avoid wrong error
...
fix https://github.com/chrislusf/seaweedfs/issues/1691
2020-12-22 02:46:24 -08:00
Chris Lu
4f31c1bb94
go fmt
2020-12-22 02:34:08 -08:00
Chris Lu
6c4f32d173
remove unused code
2020-12-22 02:33:31 -08:00
Chris Lu
90d785a15f
filer: redis, redis cluster, cassandra support super large directory
2020-12-22 02:26:05 -08:00
Chris Lu
fe46411cd4
handle directory listing edge cases
2020-12-21 23:37:43 -08:00
Chris Lu
c74bede730
handle listing directories
2020-12-21 23:19:05 -08:00
Chris Lu
adf8cb4000
filer: path-specific stores trim out common prefixes
2020-12-21 22:57:13 -08:00
Chris Lu
0823bde8d8
refactor: split into two files
2020-12-21 22:05:15 -08:00
Chris Lu
8ab70ccf76
refactor
2020-12-21 21:46:32 -08:00
Chris Lu
65cc6dc636
refactor to dedicated function
2020-12-21 21:18:34 -08:00
Chris Lu
488c2680e8
change default filer store root
2020-12-21 21:15:22 -08:00
Chris Lu
f5a19a87be
filer: report error if the path specific store is not found
...
related https://github.com/chrislusf/seaweedfs/issues/1691
2020-12-21 12:43:52 -08:00
Chris Lu
6912bf94ae
2.16
2020-12-20 15:30:12 -08:00
Chris Lu
41c0f3ad24
filer: support path-specific filer store
2020-12-19 01:27:09 -08:00
Chris Lu
2260864b45
a cleaner fix
2020-12-18 10:39:30 -08:00
Chris Lu
b7fef8f693
fix when a volume server has only hdd volumes
2020-12-18 10:34:16 -08:00
Chris Lu
0d5683fb0e
todo: load path-specific store from filer.toml
2020-12-18 03:05:42 -08:00
Chris Lu
3269fd7eaf
filer: use store by path
2020-12-18 02:57:49 -08:00
Chris Lu
e605f1e001
refactor, add getActualStore() by path
2020-12-18 02:55:00 -08:00
Chris Lu
23903aa95b
refactoring
2020-12-18 02:35:45 -08:00
Chris Lu
4e6b316913
assert FilerStoreWrapper is VirtualFilerStore
2020-12-18 02:17:06 -08:00
Chris Lu
9abf016af1
Merge branch 'master' into support_ssd_volume
2020-12-17 22:30:19 -08:00
Chris Lu
738c4fd203
fix logging when error
...
address https://github.com/chrislusf/seaweedfs/pull/1685#issuecomment-747845601
2020-12-17 19:37:46 -08:00
Chris Lu
d9e8479c06
adjust UI max count
2020-12-17 13:47:51 -08:00
Chris Lu
3cdf5945a2
adjust UI
2020-12-17 13:37:00 -08:00
Chris Lu
f696a2b2a7
assign volumes based on disk type
2020-12-17 13:25:05 -08:00
Chris Lu
daa8157fc2
Merge branch 'master' into support_ssd_volume
2020-12-17 13:05:20 -08:00
Chris Lu
e2076201d7
volume: avoid reprocessing the same volume
...
fix https://github.com/chrislusf/seaweedfs/issues/1682
2020-12-17 13:03:39 -08:00
Chris Lu
986cbdf7d9
Revert "Merge pull request #1683 from qieqieplus/master"
...
This reverts commit 8cb67952db
, reversing
changes made to 200e56215a
.
2020-12-17 12:46:42 -08:00
Chris Lu
f56e6d231e
Revert "Merge pull request #1683 from qieqieplus/master"
...
This reverts commit 8cb67952db
, reversing
changes made to 200e56215a
.
2020-12-17 12:46:20 -08:00
Chris Lu
1bf22c0b5b
go fmt
2020-12-16 09:14:05 -08:00
Chris Lu
1d88865869
passing disk type along
2020-12-16 09:10:14 -08:00
Chris Lu
23014b6810
Merge branch 'master' into support_ssd_volume
2020-12-16 08:26:51 -08:00
Chris Lu
b789767cd1
filer: return http status 499 for client closed connection
...
fix https://github.com/chrislusf/seaweedfs/issues/1684
2020-12-16 08:18:00 -08:00
chenqieqie
45f902a9b7
smaller critical section
2020-12-16 20:23:51 +08:00
chenqieqie
4e58a4f24e
fix race condition when loading volumes concurrently
2020-12-16 18:49:10 +08:00
Chris Lu
3c7f9633eb
show error a little bit more obviously
...
fix https://github.com/chrislusf/seaweedfs/issues/1678
2020-12-14 09:39:33 -08:00
Chris Lu
97dcae827b
Merge branch 'master' into support_ssd_volume
2020-12-14 01:27:31 -08:00
Chris Lu
3d47c38262
collection.delete requires _default_ as the default empty collection name
...
fix https://github.com/chrislusf/seaweedfs/issues/1677
2020-12-14 01:05:20 -08:00
Chris Lu
2e8dba571b
adjust volume server UI
2020-12-14 00:51:57 -08:00
Chris Lu
bf5873022d
Merge branch 'master' into support_ssd_volume
2020-12-14 00:32:17 -08:00
Chris Lu
b7e3ca9172
Merge pull request #1650 from kmlebedev/masterServerEnableUI
...
security master use access.ui
2020-12-14 00:31:51 -08:00
Chris Lu
021358749b
better detect IP address
...
following advice from https://github.com/chrislusf/seaweedfs/issues/1671
2020-12-14 00:30:20 -08:00
Konstantin Lebedev
c4459249ef
ui interface is always on
2020-12-14 13:24:41 +05:00
Chris Lu
5cffddebb9
add logging
2020-12-14 00:11:52 -08:00
Chris Lu
933f2f4cfd
find location with matching disk type
2020-12-13 23:39:00 -08:00
Chris Lu
94525aa0fd
allocate volume by disk type
2020-12-13 23:08:21 -08:00
Chris Lu
f6a419c26c
disk type configurable for each folder
2020-12-13 22:49:56 -08:00
Chris Lu
7c52a35942
refactor
2020-12-13 22:29:52 -08:00
Chris Lu
c45d6a865b
Merge branch 'master' into support_ssd_volume
2020-12-13 22:14:18 -08:00
Chris Lu
38fc89d041
filer store: sql update if any insert error happens
...
fix https://github.com/chrislusf/seaweedfs/issues/1673
2020-12-13 20:49:44 -08:00
Chris Lu
23280257df
sql put kv: avoid unnecessary update
2020-12-13 19:45:47 -08:00
Chris Lu
a9db24cd05
master allocate volumes if ssd type runs out
2020-12-13 19:44:57 -08:00
Chris Lu
ac4d4a65af
Update Makefile
2020-12-13 12:34:58 -08:00
Chris Lu
8baba93fce
rename parameter name to "disk"
2020-12-13 12:06:21 -08:00
Chris Lu
51eadaf2b6
rename parameter name to "disk"
2020-12-13 12:05:31 -08:00
Chris Lu
0d2ec832e2
rename from volumeType to diskType
2020-12-13 11:59:32 -08:00
Chris Lu
715b199eeb
fix tests
2020-12-13 04:14:50 -08:00
Chris Lu
66cd219b15
Merge branch 'master' into support_ssd_volume
2020-12-13 04:08:36 -08:00
Chris Lu
813453f3e1
2.15
2020-12-13 03:45:35 -08:00
Chris Lu
17372ac3da
balance ssd volumes
2020-12-13 03:40:33 -08:00
Chris Lu
d156c74ec0
volume server set volume type and heartbeat to the master
2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3
adding volume type
2020-12-13 00:58:58 -08:00
Chris Lu
16cd6fb278
fix variable name
2020-12-12 21:19:40 -08:00
Chris Lu
5c465293e9
correctly determine whether a folder is empty
...
avoid edge cases that deleting the folder if previous 32 directories are all empty
early terminate if one file is found
2020-12-12 16:19:29 -08:00
Chris Lu
14910d035c
the pagination size was too big for recursive deletion
2020-12-12 13:26:10 -08:00
Chris Lu
f930c713fc
more efficient recursion
2020-12-12 13:25:19 -08:00
Chris Lu
316d1b4e69
refactor APIs
2020-12-12 12:42:53 -08:00
Chris Lu
03637d6f57
s3: move "delete-directory-if-empty" to read time
...
move "delete-directory-if-empty" to read time instead of entry deletion time
the listing speed for a s3 bucket folder will slow down if it has many sub folders
related to 0d345ac97d
fix https://github.com/chrislusf/seaweedfs/issues/1647
fix https://github.com/chrislusf/seaweedfs/issues/1670
2020-12-12 03:38:34 -08:00
Chris Lu
37075a414d
adjust logs
2020-12-12 03:33:57 -08:00
Chris Lu
ae655033ac
adjust logging
2020-12-11 16:57:53 -08:00
Chris Lu
d986c7196d
use append time instead of filer's own modification time
...
fix https://github.com/chrislusf/seaweedfs/issues/1669
2020-12-11 16:55:18 -08:00
Chris Lu
3fedfec1e7
check cross device rename error
2020-12-10 23:50:32 -08:00
Chris Lu
c2f18a10cb
minor
2020-12-10 22:23:22 -08:00
Chris Lu
4b0c2a846b
add logs
2020-12-10 19:55:28 -08:00
Chris Lu
83078ac6ce
filer: change to /etc/seaweedfs folder on filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1666
2020-12-10 11:11:02 -08:00
Chris Lu
26731694f8
s3: use static configuration by default
...
So that users can still use the previous configuration files.
If leave it empty, s3 will try to use the version from filer
2020-12-10 00:59:04 -08:00
Chris Lu
765b3ef1be
save /etc/iam/identity.json inside filer store
2020-12-10 00:15:22 -08:00
Chris Lu
97c942b5ee
paginate through large directories
2020-12-09 23:23:38 -08:00
Chris Lu
02a160c3fd
s3: recursively iterate all sub folders
...
fix https://github.com/chrislusf/seaweedfs/issues/1656
2020-12-09 16:47:34 -08:00
Chris Lu
a9c619c270
added error handling for saving
2020-12-09 01:34:24 -08:00
Chris Lu
4211601eab
set file handle reader to nil
2020-12-08 22:48:18 -08:00
Chris Lu
b52ae9cef8
add s3 debug
2020-12-08 22:27:04 -08:00
Chris Lu
8e78187a97
add back last read chunk cache to reader and properly close the reader
2020-12-08 22:26:46 -08:00
Chris Lu
900d22c6ec
mount: avoid memory leaking read buffer
...
fix https://github.com/chrislusf/seaweedfs/issues/1654
the reader goes together with the file handle, which may stay for a long time.
2020-12-08 02:38:53 -08:00
Chris Lu
06bb7bf6c0
adding /usr/local/etc/seaweedfs for bsd style config search path
...
related to https://reviews.freebsd.org/D27391
2020-12-07 16:46:48 -08:00
Chris Lu
fce8803087
break import cycle
2020-12-07 00:29:17 -08:00
Chris Lu
606051cc79
filer: cache "/etc" filder small files to filer store
2020-12-07 00:11:35 -08:00
Chris Lu
ac22f1dd26
filer subscribe: handle rename subscription
2020-12-07 00:10:49 -08:00
Chris Lu
eed87791b7
s3: subscribe to s3.configure changes
2020-12-07 00:10:29 -08:00
Chris Lu
01e2da5782
refactoring
2020-12-06 23:16:20 -08:00
Chris Lu
77286f8bea
fix test
2020-12-06 21:56:13 -08:00
Chris Lu
57578a6cd1
change parameter help message
2020-12-06 21:54:55 -08:00
Chris Lu
78f3ab439c
refactoring
2020-12-06 21:54:34 -08:00
Chris Lu
9acda432fe
fix import cycle
2020-12-06 20:12:52 -08:00
Chris Lu
ae5eb85a06
refactoring
2020-12-06 20:05:06 -08:00
Chris Lu
5931a2f53f
filer conf: support json conf only
2020-12-06 19:47:06 -08:00
Chris Lu
4c72482536
shell: fix fs.configure
2020-12-06 19:43:31 -08:00
Chris Lu
46b91228af
2.14
2020-12-06 18:56:58 -08:00
Chris Lu
263eb29e9f
filer: add option to cache small files to filer store
2020-12-04 22:39:43 -08:00
Chris Lu
d171d9f988
volume: ensure the volume dat and idx files are always closed
...
fix https://github.com/chrislusf/seaweedfs/issues/1646
2020-12-04 21:50:26 -08:00
Konstantin Lebedev
eadbba5c95
security master use access.ui
2020-12-04 12:57:59 +05:00
Chris Lu
cc839f935d
better unit detection
2020-12-03 19:23:59 -08:00
Chris Lu
b219ccfe68
Update volume_ttl_test.go
2020-12-03 12:55:44 -08:00
Chris Lu
a83b8e7b01
ttl fix: 3y was converted into 96m
2020-12-03 10:53:29 -08:00
Chris Lu
5fb60b7135
Merge pull request #1596 from kmlebedev/store_s3cred
...
S3 credentials store in filer
2020-12-03 00:40:20 -08:00
Konstantin Lebedev
14699dfcef
use content field of entry
2020-12-02 17:19:05 +05:00
Konstantin Lebedev
a3d4b50a49
use entry content filed
2020-12-02 16:12:13 +05:00
Chris Lu
003b6245e7
fix nil
2020-12-02 00:09:19 -08:00
Chris Lu
f3bb645018
file open error
2020-12-01 23:37:49 -08:00
Chris Lu
dc0bc48257
return file open error
2020-12-01 23:36:49 -08:00
Chris Lu
45dba088cd
fix tests
2020-12-01 19:37:21 -08:00
Chris Lu
04062c56c7
webdav: improve webdav upload speed
2020-12-01 15:32:27 -08:00
Konstantin Lebedev
03620776ec
Merge branch 'upstream_master' into store_s3cred
2020-12-01 16:03:34 +05:00
Chris Lu
005a6123e9
Merge pull request #1643 from hilimd/master
...
fix tls grpc ca path
2020-11-30 17:16:46 -08:00
Chris Lu
f4abd01adf
filer: cache small file to filer store
2020-11-30 04:34:04 -08:00
Chris Lu
a9c6be5fc3
set creation time
2020-11-30 03:11:52 -08:00
Chris Lu
141ce67c09
close http request body
2020-11-30 02:45:00 -08:00
Chris Lu
0e99531dbf
2.13
2020-11-29 17:00:03 -08:00
Chris Lu
965413c21b
shell: add volume.vacuum command
2020-11-28 23:18:02 -08:00
Chris Lu
96c48bc8a8
fix test
2020-11-28 03:03:43 -08:00
Chris Lu
c57a7839ec
adjust comments
2020-11-28 03:01:03 -08:00
Chris Lu
8438fb0cc3
fix ec shard spreading error
2020-11-28 02:21:16 -08:00
Chris Lu
536202e4e0
minor
2020-11-28 02:16:06 -08:00
Chris Lu
99c4e50d3d
minor
2020-11-28 00:14:11 -08:00
Chris Lu
2c913dde04
volume: detect and drop volumes with disk IO error
...
from Jethro in slack:
is it possible to make the assign request a bit smarter? Currently I’m in the state that a disk failed but all assign request are being send to this volume. It would be cool if the master sees this and stopped using this volume.
e=HTTP(http://x:8089/913,045a782b63176edf ) not 200 but 500 Internal Server Error
Body={"size":740167,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"ee4381e202212ff3aee647704c036689"}
e=HTTP(http://x:8089/913,045a782c90240077 ) not 200 but 500 Internal Server Error
Body={"size":792779,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"c43463ccc11eb6eb2fc306f407a6a953"}
e=HTTP(http://x:8089/913,045a782e6b7901ea ) not 200 but 500 Internal Server Error
Body={"size":3962392,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"04c91198e9b276c81f11dbf189af5d28"}
2020-11-28 00:09:29 -08:00
Chris Lu
9ac4935f22
read from volume index file directly instead of open a separate file
...
fix https://github.com/chrislusf/seaweedfs/issues/1640
read from volume index file directly instead of open a separate file,
to ensure reading latest index entries.
2020-11-27 16:18:48 -08:00
Chris Lu
85554bea38
filer: readonly handle static resources
...
fix https://github.com/chrislusf/seaweedfs/issues/1641
2020-11-27 13:42:14 -08:00
Chris Lu
6d30b21b10
volume: add "-dir.idx" option for separate index storage
...
fix https://github.com/chrislusf/seaweedfs/issues/1265
2020-11-27 03:17:10 -08:00
Chris Lu
3c229eb677
refactor
2020-11-26 17:21:55 -08:00
Chris Lu
c0fb4a3766
add comments
2020-11-26 15:22:42 -08:00
Chris Lu
6783ae5f8f
rename
2020-11-26 15:19:43 -08:00
Chris Lu
983fd90010
marked as WIP
2020-11-26 15:08:52 -08:00
Chris Lu
97f706f35b
just refactoring
2020-11-26 15:08:05 -08:00
Chris Lu
b11449f955
pass in dir.idx parameter to DiskLocation
2020-11-26 14:59:03 -08:00
Chris Lu
0d345ac97d
s3: remove empty parent folder on delete
...
fix https://github.com/chrislusf/seaweedfs/issues/1637
2020-11-26 12:21:58 -08:00
Chris Lu
cc2bd97ad9
refactor
2020-11-26 11:25:56 -08:00
Chris Lu
0da7ecfd29
go fmt
2020-11-26 11:22:30 -08:00
Chris Lu
effa00ed08
refactor
2020-11-26 11:21:58 -08:00
Chris Lu
513bcd6e0d
filer: avoid duplicated FindEntry for deletion
2020-11-26 11:14:56 -08:00
Chris Lu
1ae108efca
add debug message
2020-11-26 10:41:10 -08:00
Chris Lu
a102157020
do not retry if failed with a non transport related error
2020-11-26 01:29:30 -08:00
Konstantin Lebedev
4e55baf5b1
s3 config changes
2020-11-26 03:50:53 +05:00
Konstantin Lebedev
52c8f2fc9a
s3iam test
2020-11-26 02:26:45 +05:00
Konstantin Lebedev
a26f1b2040
new pkg s3iam
2020-11-26 01:30:11 +05:00
Konstantin Lebedev
6206737df2
s3 configure
2020-11-25 21:02:31 +05:00
Chris Lu
eab53ea80d
filer leveldb store: a bit more efficient directory listing with prefix
2020-11-22 21:10:41 -08:00
Chris Lu
d3be2ceedd
2.12
2020-11-22 17:17:33 -08:00
Chris Lu
c7ebadc25d
avoid possible concurrent access inside ensureCorrectWritables()
2020-11-22 17:15:59 -08:00
hilimd
9aa990f80f
Merge pull request #39 from chrislusf/master
...
sync
2020-11-22 20:29:17 +08:00
limd
e8296104fc
fix tls grpc ca path
2020-11-22 20:27:15 +08:00
Chris Lu
92f906b6fc
remove zstd
...
fix https://github.com/chrislusf/seaweedfs/issues/1629
2020-11-21 13:06:45 -08:00
Aleksandr Demshin
176950ea78
fix typo
2020-11-20 19:50:46 +05:00
Chris Lu
7385103057
fix volume placement validating
...
fix https://github.com/chrislusf/seaweedfs/issues/1626
2020-11-20 01:12:25 -08:00
Konstantin Lebedev
27e73de797
Merge branch 'upstream_master' into store_s3cred
...
# Conflicts:
# weed/s3api/filer_util.go
2020-11-19 18:16:44 +05:00
Chris Lu
290b5e2cd0
directly delete file chunks
...
keeping current async deletions for now
2020-11-17 17:20:21 -08:00
Chris Lu
781585b195
remove unnecessary checking
2020-11-17 17:00:06 -08:00
Chris Lu
8cb8cd4cc5
add locks
2020-11-17 16:59:48 -08:00
Chris Lu
7c9e592c2d
s3: avoid nil resp when having error
...
fix https://github.com/chrislusf/seaweedfs/issues/1622
2020-11-17 11:23:17 -08:00
Chris Lu
dc304342b2
fs.configure: configurable volume growth
2020-11-17 01:00:02 -08:00
Chris Lu
b04375a3d1
use alternative assign request
2020-11-17 00:36:21 -08:00
Chris Lu
9add554feb
fail fast if user configures collection for a bucket
2020-11-16 19:57:08 -08:00
Chris Lu
6204dc6e85
add comment
2020-11-16 16:57:31 -08:00
Chris Lu
ef3e068c51
deprecating "filer.options.buckets_fsync" in filer.toml
2020-11-16 16:56:49 -08:00
Chris Lu
5f19e81dab
filer confi: support hierachical configuration
2020-11-16 16:50:12 -08:00
Chris Lu
ed1ce3f299
adjust help message
2020-11-16 09:54:26 -08:00
Chris Lu
c0d279c54e
filere.conf: prefer to use json format
2020-11-15 21:48:17 -08:00
Chris Lu
83527a8f55
adds more help message
2020-11-15 21:31:14 -08:00
Chris Lu
af658ea970
fix typo
2020-11-15 21:16:28 -08:00
Chris Lu
98827d68d9
2.11
2020-11-15 20:25:56 -08:00
Chris Lu
ee2fa14dbe
filer conf: delete location specific configuration
2020-11-15 20:15:47 -08:00
Chris Lu
2bd6fd3bbe
remove unused function
2020-11-15 20:15:07 -08:00
Chris Lu
71056dae07
fs.configure: read and local add filer configuration
2020-11-15 18:09:35 -08:00
Chris Lu
0ea5c087ce
go fmt
2020-11-15 16:59:28 -08:00
Chris Lu
95c0de285d
refactoring
2020-11-15 16:58:48 -08:00
Chris Lu
500bcab953
refactoring
2020-11-15 14:41:56 -08:00
Chris Lu
590f02179d
filer: load filer conf when starting
2020-11-15 14:06:03 -08:00
Chris Lu
0a406f652e
load filer conf and match by prefix
2020-11-15 00:26:05 -08:00
Chris Lu
68043cfcac
add reference implementation to detect create/update/delete/rename events
2020-11-14 21:21:58 -08:00
Chris Lu
0fc9ffc603
able to read filer.conf if updated
2020-11-14 21:21:20 -08:00
Chris Lu
442e092995
filer watch "/etc" folder for configuration changes
2020-11-14 14:26:08 -08:00
ruitao.liu
a9990a1dc6
adjust check bucket if exist or has access.
2020-11-13 17:13:20 +08:00
Chris Lu
e6333da65a
enable admin to access all buckets
2020-11-12 13:57:54 -08:00
Chris Lu
7094492428
fix ErrNotFound
2020-11-12 13:30:46 -08:00
Chris Lu
c6a0704019
rename function
2020-11-12 13:30:08 -08:00
Chris Lu
ed3b0f81c6
Merge pull request #1613 from taozix/master
...
check permission for bucket delete/head.
2020-11-12 13:25:28 -08:00
ruitao.liu
c4f0fd6e1b
skip if entry.Extended map is nil.
2020-11-12 17:59:31 +08:00
ruitao.liu
ab966410d2
return NoSuchBucket instead of InternalError delete non-existed bucket.
2020-11-12 16:44:16 +08:00
Chris Lu
a4dfa00030
adjust filer conf
2020-11-12 00:39:59 -08:00
Chris Lu
2251a8e472
minor
2020-11-12 00:38:58 -08:00
Chris Lu
5a16f17e47
remove unused message type
2020-11-12 00:38:23 -08:00
ruitao.liu
e06676f007
check permission for bucket delete/head.
2020-11-12 16:15:59 +08:00
Chris Lu
c32b6f4d9d
prepare for path based configuration
2020-11-11 23:46:45 -08:00
Chris Lu
0fbc27b640
Merge pull request #1612 from taozix/master
...
add owner attr for bucket.
2020-11-11 22:19:47 -08:00
ruitao.liu
d7cc0498e0
check if bucket already exists.
2020-11-12 14:11:03 +08:00
Chris Lu
827930e249
Merge pull request #1610 from kmlebedev/prefer_read_in_this_dc
...
prefer to read from volumes in this data center
2020-11-11 22:09:41 -08:00
Konstantin Lebedev
0880aff224
update dc flag desc
2020-11-12 10:07:52 +05:00
ruitao.liu
dbba8cb57e
fix if nil map in entry.
2020-11-12 11:50:19 +08:00
Konstantin Lebedev
aa30604cec
fix tests
2020-11-12 08:49:38 +05:00
Konstantin Lebedev
1eec5c8d5d
gen pb
2020-11-12 04:10:06 +05:00
Konstantin Lebedev
fc7baef5bb
fiil serverUrls sorted by data center
2020-11-12 02:13:33 +05:00
Chris Lu
745ee8d8f3
avoid unnecessary error message
...
fix https://github.com/chrislusf/seaweedfs/issues/1611
2020-11-11 13:00:05 -08:00
Chris Lu
a29abd8600
add a {X-,}Idempotency-Key header for http POST
...
fix https://github.com/chrislusf/seaweedfs/issues/1592
according to https://github.com/golang/go/issues/19943
2020-11-11 12:56:53 -08:00
Chris Lu
8708e7a64d
print out data size
2020-11-11 12:51:44 -08:00
Chris Lu
c6bd244ebd
add TODO
2020-11-11 12:51:27 -08:00
Konstantin Lebedev
b73ef6aa95
Merge remote-tracking branch 'upstream/master' into prefer_read_in_this_dc
2020-11-11 23:29:57 +05:00
Konstantin Lebedev
dc26012a3b
initial
2020-11-11 15:03:47 +05:00
Chris Lu
86cdb2a3e5
adjust logs
2020-11-11 02:01:24 -08:00
Chris Lu
73f934d5de
s3: do not close reader too early
...
fix https://github.com/chrislusf/seaweedfs/issues/1609
2020-11-11 01:49:39 -08:00
ruitao.liu
5b636b3242
Add bucket owner attr.
2020-11-11 16:20:59 +08:00
Chris Lu
15956d8a8e
2.10
2020-11-10 23:17:13 -08:00
Chris Lu
de3bdd0651
delete old volume replica
...
related to https://github.com/chrislusf/seaweedfs/issues/1607
old is:
* older compaction revision
* older modified time
* smaller volume size
2020-11-10 12:26:05 -08:00
Chris Lu
1dd3a6ac36
avoid verbose logs
2020-11-09 15:56:11 -08:00
Chris Lu
6856b0d57e
filer: add API to add/modify/delete tagging
2020-11-09 01:00:07 -08:00
Konstantin Lebedev
6cd6ff6962
avoid blank response for 404
2020-11-04 22:54:47 +05:00
Chris Lu
0dafcf1f5a
volume: detect max volume count changes based on disk usage
...
fix https://github.com/chrislusf/seaweedfs/issues/1594
2020-11-03 14:43:17 -08:00
Chris Lu
06ee199ef3
ensure entryViewCache is in sync with the chunks
2020-11-03 12:22:19 -08:00
Chris Lu
b132cb893f
in case some chunks modified earliar are added later
2020-11-03 12:22:19 -08:00
Chris Lu
0ae4d7bfdf
lock "addChunks()"
2020-11-03 12:22:19 -08:00
Konstantin Lebedev
e1190b3224
load S3 config from filer
...
https://github.com/chrislusf/seaweedfs/issues/1500
2020-11-03 21:45:56 +05:00
Chris Lu
d29b787220
store together with "Seaweed-" prefix
2020-11-03 00:21:10 -08:00
Chris Lu
5448781dfc
filer: Headers with "Seaweed-" prefix are stored as extended properties
2020-11-03 00:15:51 -08:00
Chris Lu
0ecff9521d
avoid []byte reuse
2020-11-02 23:27:54 -08:00
Chris Lu
031aa70da6
Revert "revert bytebufferpool, seems problematic"
...
This reverts commit e0fcab47c0
.
2020-11-02 19:11:25 -08:00
Chris Lu
7d5db73d98
go fmt
2020-11-02 14:21:07 -08:00
Chris Lu
a9662d44ac
avoid closing chunkSaveErrChan too early
2020-11-02 14:20:38 -08:00
Chris Lu
e0fcab47c0
revert bytebufferpool, seems problematic
2020-11-02 13:49:38 -08:00
Chris Lu
e1423a83dd
mount: clear possible leftover entry data in memory
2020-11-01 23:28:56 -08:00
Chris Lu
9b10d1ac33
2.08
2020-11-01 12:45:20 -08:00
Chris Lu
8750cac090
move to util.RetryWaitTime
2020-11-01 02:36:43 -08:00
Chris Lu
ef908e166b
break import cycle
2020-11-01 02:28:11 -08:00
Chris Lu
4609644658
mount: remove files that may still are open
...
related to https://github.com/chrislusf/seaweedfs/issues/1581
2020-11-01 01:33:26 -08:00
Chris Lu
df8d976bb0
refactoring
2020-11-01 01:58:48 -07:00
Chris Lu
7192a378cc
add retry to master clients also
2020-11-01 01:40:16 -07:00
Chris Lu
004aa9ca5d
only retry transport related errors
2020-11-01 01:21:44 -07:00
Chris Lu
f7b21973ac
mount: add retry for all operations with filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1589
2020-11-01 01:11:08 -07:00
Chris Lu
de86945aeb
go fmt
2020-10-31 16:45:38 -07:00
Chris Lu
306062b4e7
filer: add CORS support
2020-10-31 16:44:03 -07:00
Chris Lu
05acc33294
volume: add CORS support
2020-10-31 16:31:39 -07:00
Chris Lu
9708df47d1
ensure offset in the index file is positive
2020-10-31 15:50:01 -07:00
Chris Lu
10f9f1cc26
handle mknod requests
2020-10-30 23:51:32 -07:00
Chris Lu
8826601be1
mount: optional limit for the number of concurrent writers
2020-10-30 21:22:20 -07:00
Chris Lu
be95f68ca7
scaffold for sftpd
2020-10-30 20:57:08 -07:00
Konstantin Lebedev
d2a4ffdd60
glog
2020-10-31 03:21:54 +05:00
Konstantin Lebedev
46303c36bf
When the volume server is stopped, the master server immediately sees the deletion of volumes
2020-10-31 02:48:25 +05:00
Chris Lu
eb32af25d4
make space for the QR code
2020-10-30 10:55:31 -07:00
Chris Lu
7ecbb4b3c9
clean up
2020-10-30 10:46:31 -07:00
Kenny
9cc6575ab5
Update filer_copy.go
...
修改 weed filer.copy 上传目录时文件名变子目录的问题
2020-10-30 18:16:49 +08:00
Chris Lu
c057dd1f64
add back upload retries
2020-10-30 02:16:34 -07:00
Chris Lu
19098f2c2d
make a copy of the filer.entry for most read operations
2020-10-30 01:23:26 -07:00
Chris Lu
6135cbaa02
clear file.entry
2020-10-30 01:23:26 -07:00
Chris Lu
02dc51b1ed
properly clear out the file.entry
2020-10-29 23:31:06 -07:00
Chris Lu
6560ac6466
volume loading: trim out unreachable idx file content
...
fix https://github.com/chrislusf/seaweedfs/issues/1583
2020-10-29 22:25:23 -07:00
Chris Lu
0b68b68ec4
reload entry only when it is a hard link
...
fix https://github.com/chrislusf/seaweedfs/issues/1581
2020-10-29 16:32:45 -07:00
Chris Lu
47047516d6
Revert "increase default volume file size limit to 1024"
...
This reverts commit 0983060a
I am confused why this was changed to 1024 by myself. 1GB should be too large for most cases.
2020-10-29 15:46:29 -07:00
Chris Lu
6f8b426f4f
Merge pull request #1578 from taozix/master
...
S3 bucket list, response with uploaded storageclass.
2020-10-29 08:32:27 -07:00
ruitao.liu
22a9ea0512
adjust s3 header file.
2020-10-29 16:05:40 +08:00
Chris Lu
c6d9974fe7
server: Add option to enable/disable volume server
...
fix https://github.com/chrislusf/seaweedfs/issues/1572
2020-10-29 00:24:18 -07:00
ruitao.liu
b917be7955
S3 bucket list, response with uploaded storageclass.
2020-10-29 14:57:19 +08:00
Konstantin Lebedev
918546bbdb
add WaitForReady to grpc DialOption
2020-10-29 08:29:04 +05:00
Konstantin Lebedev
fab01f9d8d
add readonly
2020-10-28 23:17:03 +05:00
Konstantin Lebedev
884db215a1
add shell command volume mark writable
2020-10-28 22:47:09 +05:00
Konstantin Lebedev
1ad1b8c4f6
collection Volume ReadOnly Count with detailed status
2020-10-28 17:38:26 +05:00
Chris Lu
326fcdd86b
Merge pull request #1569 from taozix/master
...
save s3 metadata to filer.
2020-10-28 03:27:24 -07:00
ruitao.liu
2dcc178d0a
save s3 metadata to filer.
2020-10-28 18:16:05 +08:00
Chris Lu
744c5594da
volume: automatically trim out unreachable entries
2020-10-28 01:14:39 -07:00
Chris Lu
3e925faddd
trim out name extension
2020-10-28 01:14:05 -07:00
shibinbin
a67eb1afa1
fix: restart volumeserver sometime occurred out of memory error
2020-10-28 12:03:06 +08:00
Chris Lu
53c3aad875
volume: add a note file to avoid incomplete volume files
...
fix https://github.com/chrislusf/seaweedfs/issues/1567
2020-10-27 15:56:49 -07:00
Chris Lu
6da87720eb
shorter tag prefix
2020-10-27 15:33:26 -07:00
Chris Lu
24bf142596
copy large file first
2020-10-27 15:31:15 -07:00
Chris Lu
432c6f8db3
Merge pull request #1566 from taozix/master
...
return x-amz-tag-count header when GET object.
2020-10-27 13:16:58 -07:00
Chris Lu
06c15ab35c
volume: add special handling for .dat larger than 32GB
2020-10-27 13:11:56 -07:00
ruitao.liu
72f6b77dea
fix tag prefix.
2020-10-27 18:01:37 +08:00
ruitao.liu
8766ca1b95
rename s3 tag prefix.
2020-10-27 17:33:24 +08:00
ruitao.liu
19026ae55d
return x-amz-tag-count header when GET object.
2020-10-27 16:49:31 +08:00
Chris Lu
44921220b0
2.07
2020-10-25 22:03:46 -07:00
Chris Lu
e71463a9eb
mount: invalide file cache when metadata is changed
2020-10-25 19:24:15 -07:00
Chris Lu
e219c57849
passing full path when assign volume locations
2020-10-25 15:46:29 -07:00
Chris Lu
f375b93aef
renaming
2020-10-25 15:32:43 -07:00
Chris Lu
63b0fb54f7
minor
2020-10-25 14:15:53 -07:00
Chris Lu
27ba0a2be3
2.06
2020-10-25 12:36:47 -07:00
Chris Lu
147d4d23b6
filer.copy: automatically use the right bucket
2020-10-25 11:21:33 -07:00
Chris Lu
91fd311f7a
fix logic to read entry or not
2020-10-25 10:33:51 -07:00
Chris Lu
10a4a628e9
refresh cached file entry from sync metadata updates
2020-10-25 09:42:50 -07:00
Konstantin Lebedev
2fb1fce8a0
avoid old values when missing read only flags
2020-10-25 16:21:40 +05:00
Chris Lu
da36abf033
go fmt
2020-10-24 20:12:04 -07:00
Chris Lu
16fae84414
fix format error
2020-10-24 20:11:31 -07:00
Chris Lu
b7b1eb51b1
default to 2 for better performance in common computer hardware
2020-10-24 20:08:26 -07:00
Chris Lu
79690b9cd8
revert changes causing wrong data
2020-10-24 20:07:19 -07:00
Chris Lu
9104cfa744
reduce locks
2020-10-24 19:40:35 -07:00
Chris Lu
e9d40b80b2
less verbose logs
2020-10-24 09:42:54 -07:00
Chris Lu
6c4fb243a7
remove limits to concurrent writers
2020-10-24 02:05:05 -07:00
Chris Lu
e0002f8dd7
check existing volumes for writable status
2020-10-24 01:34:31 -07:00
Chris Lu
19772d70d7
print for debugging
2020-10-24 00:12:02 -07:00
Chris Lu
5b92dfe302
adjust log level
2020-10-23 23:58:54 -07:00
Chris Lu
fb124f2d4a
conditionally display volume server public url link
2020-10-23 23:58:46 -07:00
Chris Lu
1b17f71939
adjust election timeout to 10 seconds
2020-10-23 23:06:44 -07:00
Chris Lu
ef9c32ea0d
adjust compression threshold
2020-10-23 23:05:55 -07:00
Chris Lu
0cea84e7ab
make it easy with setting up multiple masters in a clcuster
2020-10-23 18:18:46 -07:00
Chris Lu
fc689319ae
add volume public url
2020-10-23 18:03:23 -07:00
Chris Lu
71624858af
avoid reusing cached []byte
...
this logic is just to ensure 100% correct.
fuse message, data+message hearder > 512 will not be cached.
2020-10-23 11:31:57 -07:00
Chris Lu
29c9fa2ef2
byte buffer for uploading
2020-10-23 11:29:51 -07:00
Chris Lu
2579edbc60
fix bug found by tests
2020-10-23 10:05:40 -07:00
Chris Lu
e0c8507d93
memory pool to reduce memory allocation
2020-10-23 00:41:40 -07:00
Chris Lu
b81359823f
postgres: support empty user
2020-10-22 14:27:47 -07:00
Chris Lu
11716fbf6f
make reading error more obvious
2020-10-22 13:33:45 -07:00
Chris Lu
d75bc62196
to read files that are quickly changing
2020-10-22 12:59:15 -07:00
Chris Lu
575d7952a1
add available resource stats
...
fix https://github.com/chrislusf/seaweedfs/issues/1555
2020-10-22 09:13:47 -07:00
Chris Lu
5179e559f7
skip empty logs
2020-10-22 00:35:48 -07:00
Chris Lu
de044e466e
Merge pull request #1553 from kmlebedev/multi_domainname
...
Multi domainname
2020-10-21 23:56:03 -07:00
Chris Lu
e73d6c9526
able to print partial file id
...
address https://github.com/chrislusf/seaweedfs/issues/1552
2020-10-21 23:48:07 -07:00
Konstantin Lebedev
a9a7005687
set desc of option
2020-10-22 11:23:00 +05:00
Chris Lu
720b1d9b88
adding locking to avoid nil VolumeLocationList
...
fix panic: runtime error: invalid memory address or nil pointer dereference
Oct 22 00:53:44 bedb-master1 weed[8055]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x17658da]
Oct 22 00:53:44 bedb-master1 weed[8055]: goroutine 310 [running]:
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLocationList).Length(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_location_list.go:35
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).enoughCopies(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:376
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).ensureCorrectWritables(0xc000111d50, 0xc000b55438)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:202 +0x5a
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*Topology).SyncDataNodeRegistration(0xc00042ac60, 0xc001454d30, 0x1, 0x1, 0xc0005fc000, 0xc00135de40, 0x4, 0xc00135de50, 0x10, 0x10d, ...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/topology.go:224 +0x616
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/server.(*MasterServer).SendHeartbeat(0xc000162700, 0x23b97c0, 0xc000ae2c90, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/server/master_grpc_server.go:106 +0x325
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/pb/master_pb._Seaweed_SendHeartbeat_Handler(0x1f8e7c0, 0xc000162700, 0x23b0a60, 0xc00024b440, 0x3172c38, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/pb/master_pb/master.pb.go:4250 +0xad
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0xc0001fea80, 0x311fec0, 0x0, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1329 +0xcd8
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).handleStream(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1409 +0xc5c
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0001ce8b0, 0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:746 +0xa5
Oct 22 00:53:44 bedb-master1 weed[8055]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:744 +0xa5
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Failed with result 'exit-code'.
2020-10-21 23:15:48 -07:00
Chris Lu
2bc05cfbc8
fix typo
2020-10-21 22:26:41 -07:00
Chris Lu
4ff2ceee33
UI fix on rendering EC volumes
...
addressing UI problem with https://github.com/chrislusf/seaweedfs/issues/1551
2020-10-21 22:05:58 -07:00
Chris Lu
f5ee19e0db
shortcut for appending operations
2020-10-21 20:44:01 -07:00
Chris Lu
b5917bf846
clean up
2020-10-21 19:31:26 -07:00
Chris Lu
9884dfc369
use bytebuffer for interval list
2020-10-21 19:29:51 -07:00
Chris Lu
bd103c143a
add lock for vidCache
2020-10-21 19:28:59 -07:00
Konstantin Lebedev
05b5f12f2e
multiplate DomainNames through comma
2020-10-21 20:48:51 +05:00
Chris Lu
81cf8d04df
minor
2020-10-21 02:42:23 -07:00
Chris Lu
d90e7ee3cc
adjust log level
2020-10-21 02:17:40 -07:00
Chris Lu
3bf0116de1
mount: less channel waiting
2020-10-21 02:16:21 -07:00
Chris Lu
c31b254248
mount: shortcut when there is only one chunk
2020-10-20 23:48:29 -07:00
Chris Lu
f64252023e
Revert "a better byte buffer pool"
...
This reverts commit 59c6422777
.
2020-10-20 23:21:54 -07:00
Chris Lu
ad652ffb97
avoid close closed channel
2020-10-20 22:54:34 -07:00
Chris Lu
20442dc1bc
avoid extra data copying
2020-10-20 22:54:21 -07:00
Chris Lu
59c6422777
a better byte buffer pool
2020-10-20 22:53:58 -07:00
Chris Lu
6c92fa349d
filer: support default rack
...
fix https://github.com/chrislusf/seaweedfs/issues/1546
2020-10-20 17:41:39 -07:00
Chris Lu
b3aa2fab9a
s3: report error to s3 when updating an object but it is already a directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1545
2020-10-20 10:25:16 -07:00
Chris Lu
410b818aa7
master: avoid timer leakage
2020-10-19 14:24:57 -07:00
Chris Lu
69f0da5d73
mount: pass along replication and collection parameters
2020-10-19 00:12:42 -07:00
Chris Lu
c0ab458671
report nil instead of EOF for empty files
...
related to https://github.com/chrislusf/seaweedfs/issues/1541
2020-10-17 11:03:46 -07:00
Chris Lu
898db14729
2.05
2020-10-17 02:42:33 -07:00
Chris Lu
09bab17aff
mount: avoid "send on closed channel"
2020-10-16 12:54:23 -07:00
Chris Lu
9d80a3428c
add debug_webdav
2020-10-16 12:53:56 -07:00
Chris Lu
37f165d743
webdav: return io.EOF when at end of a file
...
fix https://github.com/chrislusf/seaweedfs/issues/1344
2020-10-16 12:53:37 -07:00
Chris Lu
68d39c86f1
mysql, postgres, cassandra: change kv key to base64 encoding
...
The exisitng key-value operation for stores using mysql, postgres, and maybe cassandra are already broken.
The kv is used to store hardlink, filer store signature and replication progress.
So users using hardlink and also uses mysql, postgres, or cassandra will have broken hard links.
Users using filer.sync will need to re-sync the files.
2020-10-16 11:10:12 -07:00
Chris Lu
ee1fc6558a
refactor
2020-10-16 11:02:44 -07:00
Chris Lu
06e6341097
set initial write time to avoid race condition with delayed chunks
2020-10-15 11:08:45 -07:00
Chris Lu
d598c47136
log errors
2020-10-15 11:01:43 -07:00
Chris Lu
5039aff310
release the goroutine to read errors
2020-10-15 10:59:18 -07:00
Chris Lu
ace0ea3d28
s3: avoid duplicated bucket
2020-10-15 10:52:20 -07:00
Konstantin Lebedev
dc2e13092d
add number of read only volumes metric
2020-10-15 15:48:40 +05:00
Chris Lu
e714c28a02
mount: async write file chunk
2020-10-14 23:28:03 -07:00
Chris Lu
c95d3d9dff
adjust tests
2020-10-14 12:27:52 -07:00
Chris Lu
93bcf56514
file read report EOF
...
fix https://github.com/chrislusf/seaweedfs/issues/1344
2020-10-14 12:18:24 -07:00
Chris Lu
1069b325dd
shell: volumeServer.evacuate adds printout for ec volumes
2020-10-13 20:26:03 -07:00
Chris Lu
0542911e29
go fmt
2020-10-13 19:50:46 -07:00
Chris Lu
58fa506491
minor
2020-10-13 19:50:22 -07:00
Chris Lu
28d4e1a51b
mount: retry for directory listing with filer
...
related to https://github.com/chrislusf/seaweedfs/issues/1530
2020-10-13 19:49:52 -07:00
Chris Lu
c127da1219
filer: linearize timeout for large chunk of data
2020-10-13 14:04:46 -07:00
Chris Lu
aac4cb1f0c
adds errror on read and write
2020-10-13 13:53:34 -07:00
Chris Lu
9b4f7fed14
mount: report filer IO error
...
related to https://github.com/chrislusf/seaweedfs/issues/1530
2020-10-13 11:21:13 -07:00
Chris Lu
3f7d1d1bf1
Only wait on retryable requests
2020-10-13 00:29:46 -07:00
Chris Lu
b18f21cce1
mount: fix bound tree with filer.path
...
fix https://github.com/chrislusf/seaweedfs/issues/1528
2020-10-12 21:58:37 -07:00
Chris Lu
f022aff289
add back http.StatusPartialContent
...
revert e7c04af1d0
2020-10-12 12:26:25 -07:00
Chris Lu
bbd0afd37e
2.04
2020-10-11 21:25:30 -07:00
Chris Lu
7704469d29
mount: outsideContainerClusterMode changed to use volume server publicUrl
2020-10-11 20:42:15 -07:00
Chris Lu
723ae11db4
refactoring in order to adjust volume server url later
2020-10-11 20:15:10 -07:00
Chris Lu
d155f907c2
mount: configurable read wait time
2020-10-10 20:09:43 -07:00
Chris Lu
9b0e8ef026
filer: added QR code to transfer files with mobile devices
2020-10-10 18:00:21 -07:00
Chris Lu
8a52379ecb
add retry if volume can not be found
2020-10-10 16:02:39 -07:00
Chris Lu
70af0ec24c
filer: fix hanging on read
2020-10-10 16:02:10 -07:00
Chris Lu
cff8bb6554
return proper error
2020-10-10 15:43:22 -07:00
Chris Lu
6da8eef54f
fix logic error
2020-10-09 00:01:47 -07:00
Chris Lu
5d01dd28a6
add logging
2020-10-08 23:38:00 -07:00
Chris Lu
8d34eb0050
mount:exponentially backoff if read error for about 10 minutes
2020-10-08 23:31:26 -07:00
Chris Lu
6e1f936efd
refactoring
2020-10-08 23:19:42 -07:00
Chris Lu
b2ee5873fb
fix error not being returned
2020-10-08 23:19:20 -07:00
Chris Lu
b067853162
mount: ignore recursion error
...
fix https://github.com/chrislusf/seaweedfs/issues/1514
the host OS should manage recursive deletion.
2020-10-08 21:55:55 -07:00
Chris Lu
e1a8e3f900
fix logging
2020-10-08 18:40:38 -07:00
Chris Lu
8676db0f6a
adjust logging
2020-10-08 18:33:37 -07:00
Chris Lu
bd8b9b0c2e
some changes to prepare fix after failed reads
2020-10-08 18:33:06 -07:00
Chris Lu
78859e35fc
s3: adjust config examples
2020-10-08 10:12:09 -07:00
Chris Lu
9e7a2772b1
s3: only admin can list all buckets
2020-10-08 10:12:09 -07:00
Chris Lu
eed492b73b
randomize file locations
2020-10-07 23:58:32 -07:00
Chris Lu
ec08a1670b
volume: avoid hanging when stopping a volume server
2020-10-07 23:48:24 -07:00
Chris Lu
62d3d3aea0
go fmt
2020-10-07 23:30:54 -07:00
Chris Lu
e91b9c85a2
s3: Added support for "List" action in weed s3 -config=... in the config file.
...
fix https://github.com/chrislusf/seaweedfs/issues/1511
2020-10-07 23:22:35 -07:00
Chris Lu
a8624c2e4f
read from alternative replica
...
related to https://github.com/chrislusf/seaweedfs/issues/1512
2020-10-07 22:49:04 -07:00
Chris Lu
da4edf3651
master: check peers for existing leader before starting a leader election
...
fix https://github.com/chrislusf/seaweedfs/issues/1509
2020-10-07 01:25:39 -07:00
Chris Lu
c543762e23
volume: auto expands -max setting
...
I have tried to run weed volume but got errors...
cmd:
weed volume -max=0 -mserver="127.0.0.1:9333" -port=8080 -dir="/var/www/d1/sfs,/var/www/d2/sfs,/var/www/d3/sfs"
log:
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d1/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d2/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d3/sfs Permission: -rwxr-xr-x
F1006 15:09:26 17040 volume.go:142] 3 directories by -dir, but only 1 max is set by -max
What am I doing wrong? Thx.
2020-10-06 09:05:30 -07:00
Chris Lu
36492c47ec
adjust
2020-10-05 14:06:18 -07:00
Chris Lu
8f8e9ddbde
adjust API
2020-10-05 09:47:07 -07:00
Konstantin Lebedev
e4f2d9eb4a
We return etag using the same algorithm as aws s3
...
https://teppen.io/2018/06/23/aws_s3_etags/
2020-10-05 14:43:32 +05:00
Chris Lu
4fc673341f
2.03
2020-10-04 21:58:22 -07:00
Chris Lu
6b591b02af
adjust log level
2020-10-04 16:21:43 -07:00
Устюжанин Антон Александрович
702b1cb876
fix: remove deleted peers if resumeState = true
2020-10-04 21:56:17 +05:00
Chris Lu
d522df0ecd
adjust log level
2020-10-04 01:35:32 -07:00
Chris Lu
666859f7cd
mount: read prefetching
2020-10-04 01:31:04 -07:00
Chris Lu
bc47835997
mount: avoid duplicated reads
2020-10-03 20:16:42 -07:00
Chris Lu
8d65ad1444
Revert "mount: adds read prefetching"
...
This reverts commit 692f0614d1
.
2020-10-03 19:40:56 -07:00
Chris Lu
692f0614d1
mount: adds read prefetching
2020-10-03 16:50:09 -07:00
Chris Lu
707936f482
re-enable caching larger than 16MB
...
revert 62ce85610e
2020-10-03 14:12:38 -07:00
Chris Lu
2c20ef72ae
fix starting problem
2020-10-03 13:56:38 -07:00
Chris Lu
d77e3c96e9
FUSE: use the mount directory permissions
2020-10-03 13:37:35 -07:00
Chris Lu
95ffed87c0
Merge pull request #1505 from ustuzhanin/remove_raft_state
...
Resume raft state
2020-10-03 10:38:47 -07:00
Устюжанин Антон Александрович
6fbfe73683
fix: resumeState
2020-10-03 22:16:47 +05:00
Устюжанин Антон Александрович
dc31b19469
fix: restore raft state
2020-10-03 14:03:41 +05:00
Chris Lu
f781cce500
s3: support object tagging
...
* GetObjectTagging
* PutObjectTagging
* DeleteObjectTagging
2020-10-02 22:21:51 -07:00
Устюжанин Антон Александрович
8c82fb7e5f
fix: restore raft state
2020-10-02 23:01:20 +05:00
Chris Lu
9ab98fa912
s3 metrics adjust the label
2020-10-01 07:17:47 -07:00
Chris Lu
f1e879fe7a
2.02
2020-10-01 07:15:49 -07:00
Chris Lu
a1c01d716b
volume: avoid deadlock when deleting volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1501
2020-10-01 07:10:03 -07:00
Konstantin Lebedev
86329bbf2b
label name is statusCode
2020-10-01 01:22:38 +05:00
Konstantin Lebedev
68463e92c1
add status code in S3RequestCounter
2020-10-01 00:59:39 +05:00
Chris Lu
79ab10e300
adjust help message
2020-09-30 09:32:00 -07:00
Chris Lu
55cb68c09c
filer: replication follows master config if not specified
2020-09-30 09:15:55 -07:00
Chris Lu
b9887504e8
fix test
2020-09-27 23:19:50 -07:00
Chris Lu
f46eae284e
adjust for test
2020-09-27 23:08:11 -07:00
Chris Lu
75aca5e13d
2.01
2020-09-27 23:02:41 -07:00
Chris Lu
e6552b5e1e
filer: able to start s3 together
2020-09-27 23:00:43 -07:00
Chris Lu
474e2b6ac3
add a hard link marker to 16byte + maker, for future extensions
2020-09-27 22:38:30 -07:00
Chris Lu
c49e2bb9a3
adjust
2020-09-27 12:07:45 -07:00
Chris Lu
62ce85610e
skip caching too large chunks
2020-09-27 11:58:48 -07:00
Chris Lu
9ad2dcca2b
more tests
2020-09-27 11:42:51 -07:00
Chris Lu
e43d86c796
fix pre allocated volume size
2020-09-27 10:58:19 -07:00
Chris Lu
31fc7bb2e1
refactor
...
adjust for faster test
2020-09-27 10:41:29 -07:00
Chris Lu
a37535cd9f
avoid non utf-8 in filename
...
fix https://github.com/chrislusf/seaweedfs/issues/1493
2020-09-26 00:13:39 -07:00
Chris Lu
41be7a4c8f
filer: upload also set file size
...
ensure works same as fuse mount
2020-09-25 12:02:06 -07:00
Chris Lu
35f6518c36
weed upload: usePublicUrl was not being used on big files
...
fix https://github.com/chrislusf/seaweedfs/issues/1492
2020-09-25 02:41:38 -07:00
Chris Lu
22049dc2e6
Merge pull request #1491 from hilimd/master
...
s3: fix get object metadata problem
2020-09-25 02:35:00 -07:00
limd
402aef8f30
s3:
...
1.fix spark reading S3 directory wildcard problem
2.fix the problem of the spark history service writing S3 directory
reference git revsion number: b41b7ea4d0
2020-09-25 14:37:02 +08:00
Chris Lu
8516517c48
filer store: Cassandra supports username/password
2020-09-24 21:31:06 -07:00
limd
ed7816681a
Add cassandra authenticator mode
2020-09-25 11:22:47 +08:00
limd
370a98cf6b
code style
2020-09-25 11:19:42 +08:00
limd
aee3fd08b9
code style
2020-09-25 11:04:32 +08:00
Chris Lu
043b063136
Merge pull request #1482 from hilimd/master
...
Fix: s3 delete object
2020-09-24 18:21:34 -07:00
limd
48c578410f
mount: rollback default value
2020-09-25 09:18:52 +08:00
Chris Lu
dbf5327b98
s3: handle response-content-encoding
...
fix https://github.com/chrislusf/seaweedfs/issues/1487
2020-09-24 18:09:52 -07:00
Chris Lu
7726965a47
minor
2020-09-24 11:20:12 -07:00
Chris Lu
0790c6d605
fix empty hard link id
2020-09-24 11:16:43 -07:00
Chris Lu
1012df7bb5
switch hardlink id from int64 to bytes
2020-09-24 11:11:42 -07:00
Chris Lu
4856bce0ee
adjust for metrics port
2020-09-24 10:21:23 -07:00
Chris Lu
d2d3aec3e1
consolidate to one metricsPort in "weed server" mode
2020-09-24 09:55:02 -07:00
Chris Lu
2e9099369e
Merge pull request #1489 from kmlebedev/promhttp
...
Promhttp
2020-09-24 09:50:48 -07:00
Chris Lu
2e7c361a0d
hardlink deletion factors in hardlink counter
2020-09-24 09:43:52 -07:00
Chris Lu
1295347958
adjust hardlink update
...
simplify logic, pass entity content directly to hard link. The "weed mount" handles the logic to calculate hard link counter.
2020-09-24 09:43:00 -07:00
Konstantin Lebedev
98e9de6e11
fix style
2020-09-24 17:48:39 +05:00
Konstantin Lebedev
324e44d4b3
add start metrics server
2020-09-24 17:45:39 +05:00
limd
cbca14edc5
mount: fix k8s pvc and os mount directory permission bug
2020-09-24 18:07:16 +08:00
Chris Lu
5e239afdfc
hardlink works now
2020-09-24 03:06:48 -07:00
limd
59e91e9c7e
mount: fix k8s pvc mount directory permission
2020-09-24 10:25:45 +08:00
Chris Lu
c7d7b1a0f6
Merge pull request #1485 from LIBA-S/fix_oversized
...
Correct the oversized state of volume after compaction
2020-09-23 19:24:30 -07:00
Chris Lu
b80a2b3cc9
Merge pull request #1486 from LIBA-S/fix_race_condition
...
Fix a race condition when handle VolumeLocationList
2020-09-23 19:24:16 -07:00
LIBA-S
eecd6b5d35
Fix a race condition when handle VolumeLocationList
2020-09-23 20:56:51 +08:00
LIBA-S
0157798ebf
Correct the oversized state of volume after compaction
2020-09-23 20:27:42 +08:00
limd
9b21ec27e4
mount: auto created dir set corrct umask fix bug
2020-09-23 18:35:37 +08:00
hilimd
0db149fb5f
Merge pull request #18 from chrislusf/master
...
sync
2020-09-23 17:32:05 +08:00