Chris Lu
f2723c1bc8
do not idx file format
...
revert c9ab8d05fa
2020-09-12 12:42:36 -07:00
Chris Lu
ba984a4e29
1.98
2020-09-12 04:13:02 -07:00
Chris Lu
446e476a11
go fmt
2020-09-12 04:08:03 -07:00
Chris Lu
ea26a98753
volume: validate volume correctness if last entry is a deletion
2020-09-12 04:07:04 -07:00
Chris Lu
d15682b4a1
shell: volume.balance plan by ratio of fullness
2020-09-12 04:06:26 -07:00
Chris Lu
c0ee78d2fa
adjust make file
2020-09-12 04:05:42 -07:00
Chris Lu
cd9b89ba55
reduce default wait time to 10s before shutting down
2020-09-12 04:05:33 -07:00
Chris Lu
e2c741f76f
adjust replica placement after move
2020-09-12 01:01:19 -07:00
Chris Lu
2a0925590c
filer: etcd store avoid read with nil option
...
fix https://github.com/chrislusf/seaweedfs/issues/1463
2020-09-11 15:29:45 -07:00
Chris Lu
3984c3962f
add comment
2020-09-11 15:07:19 -07:00
Chris Lu
baa6bdf4d4
s3: listMultipartUploads fix output format
2020-09-11 15:04:01 -07:00
Chris Lu
3eda8d6dfc
s3: ListParts output xml format
...
fix https://github.com/chrislusf/seaweedfs/issues/1461
2020-09-11 14:53:50 -07:00
Chris Lu
ab201c2798
1.97
2020-09-11 13:47:50 -07:00
Chris Lu
9d4bdfcfdf
fix volume integrity checking
2020-09-11 11:34:10 -07:00
Chris Lu
1b8094ef75
weed export: export deleted files
2020-09-11 02:05:14 -07:00
Chris Lu
12a8f5294d
test for multi dc replication
2020-09-11 01:21:17 -07:00
Chris Lu
e60b2117c3
shell: volume balance follows replica placement
2020-09-11 00:29:25 -07:00
Chris Lu
89a62e8007
refactoring
2020-09-10 23:05:00 -07:00
Chris Lu
19537c9d21
1.96
2020-09-10 19:52:07 -07:00
Chris Lu
eaf9fdde99
avoid sharing context over separate goroutine
2020-09-10 19:51:03 -07:00
Chris Lu
cc5fe6f5ee
handle special characters in html link
2020-09-10 19:46:02 -07:00
Chris Lu
401ccf1509
fix test
2020-09-10 15:13:23 -07:00
Chris Lu
c9ab8d05fa
fixes for reading deleted fid
2020-09-10 14:42:52 -07:00
ruitao.liu
5b0676049a
change elastic initialize process similar as others.
2020-09-10 23:35:20 +08:00
Chris Lu
660d7c0edd
1.95
2020-09-10 01:32:05 -07:00
Chris Lu
0d9b858cfb
Merge pull request #1455 from taozix/master
...
add more basic elastic options.
2020-09-10 01:31:17 -07:00
ruitao.liu
9be4e97625
change logs print format.
2020-09-10 16:30:15 +08:00
ruitao.liu
719dc43af1
modify elastic urls from string to array.
2020-09-10 16:24:09 +08:00
ruitao.liu
72f9d7f047
use util to generate md5.
2020-09-10 16:11:18 +08:00
ruitao.liu
6a5b38c0d4
fix elastic kv ops.
2020-09-10 15:59:16 +08:00
ruitao.liu
3f7fbfddca
add more basic elastic options.
2020-09-10 14:22:07 +08:00
Chris Lu
b183ae54b5
master: changing 301 to 308 on redirect to volume servers
...
fix https://github.com/chrislusf/seaweedfs/issues/1454
2020-09-09 22:36:10 -07:00
Chris Lu
47b3f932e4
watch: adjust output format
2020-09-09 22:34:48 -07:00
Chris Lu
daf0a449f7
properly cancel context for streaming grpc
2020-09-09 12:07:15 -07:00
Chris Lu
7f69acd1f2
sync pprof
2020-09-09 11:33:52 -07:00
Chris Lu
387ab6796f
filer: cross cluster synchronization
2020-09-09 11:21:23 -07:00
Chris Lu
4fc0bd1a81
return http response directly
2020-09-09 03:53:09 -07:00
Chris Lu
5e13bc878c
adjust log level
2020-09-09 03:49:26 -07:00
Chris Lu
d1b816212f
return nil if no need to do anything
2020-09-07 16:03:05 -07:00
Chris Lu
64a621bcc8
shell: volume.fix.replication also purge over replicated volumes
2020-09-07 16:00:10 -07:00
Chris Lu
c18ea21f7a
1.94
2020-09-07 13:13:53 -07:00
Chris Lu
18b98fdb72
fix test
2020-09-07 12:57:38 -07:00
Chris Lu
44b3f2efc8
filer: Elastic Search return correct kv error
...
fix https://github.com/chrislusf/seaweedfs/issues/1452
2020-09-07 12:44:51 -07:00
Chris Lu
d80538a187
refactoring
2020-09-07 12:35:02 -07:00
Chris Lu
1a09bc43d1
refactor
2020-09-07 11:31:33 -07:00
Chris Lu
2b643f477d
typo
2020-09-06 12:47:55 -07:00
Chris Lu
05034aade5
printout over replicated locations
2020-09-06 12:44:02 -07:00
Chris Lu
432fe3d926
1.93
2020-09-06 12:12:09 -07:00
Chris Lu
d8af6f62df
adjust logging
2020-09-06 12:10:40 -07:00
Chris Lu
07f32feed0
print sync progress
2020-09-06 00:50:51 -07:00
Chris Lu
0af6252b2d
fix compilation
2020-09-06 00:50:38 -07:00
Chris Lu
ac494ff5e8
Update meta_aggregator.go
2020-09-06 00:29:16 -07:00
Chris Lu
c9f8f25ba5
read peer filer from start
2020-09-06 00:12:41 -07:00
Chris Lu
30dc365cbd
add self filer
2020-09-06 00:11:46 -07:00
Chris Lu
8acd714600
apply meta changes only if store is different
2020-09-05 23:25:03 -07:00
Chris Lu
d741ed66db
check whether shares the same filer store
2020-09-05 23:02:52 -07:00
Chris Lu
bba90ff3c8
read filer signature
2020-09-05 22:52:15 -07:00
Chris Lu
71b0e256c7
filer: setOrLoadFilerStoreSignature
2020-09-05 14:08:59 -07:00
ruitao.liu
450cf07505
skip the index that for kv usage.
2020-09-04 21:49:03 +08:00
ruitao.liu
1384ff9a2f
1.split kv in one file.
...
2.disable query for kv in es index.
2020-09-04 17:34:26 +08:00
ruitao.liu
83080b5e03
ES backended filer support kv ops.
2020-09-04 15:40:13 +08:00
Chris Lu
46f65a84a6
filer: elastic7 adjust default value
2020-09-03 20:12:38 -07:00
Chris Lu
f815fce81b
Merge pull request #1446 from taozix/master
...
new filer store option to es7.
2020-09-03 20:10:23 -07:00
Chris Lu
44b4ebf0ff
filer: Redis cleanly delete directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1448
2020-09-03 19:17:39 -07:00
Chris Lu
b8f32bcab9
filer: compress stored metadata
2020-09-03 11:00:20 -07:00
Chris Lu
f76a2b2c8a
printout meta data size
2020-09-03 09:51:21 -07:00
ruitao.liu
798280e980
change filer2 to filer.
2020-09-03 17:05:26 +08:00
ruitao.liu
a93d27d1e8
new filer option to es v7.
2020-09-03 16:34:58 +08:00
Chris Lu
1d56ea24ef
fix
2020-09-03 00:08:37 -07:00
Chris Lu
7e1aad0b54
mount: map uid/gid between local and filer
2020-09-03 00:07:22 -07:00
Chris Lu
68e878adb5
fix formatting
2020-09-02 21:42:12 -07:00
Chris Lu
0d99a5da3c
c*: support kv operations
2020-09-02 18:39:24 -07:00
Chris Lu
645a4af3db
mongodb: support kv operations
2020-09-02 17:19:14 -07:00
Chris Lu
ecaa30c408
better error message
2020-09-02 17:17:44 -07:00
Chris Lu
7c770b727c
Update abstract_sql_store_kv.go
2020-09-02 10:13:48 -07:00
Chris Lu
05a6e2dc7f
sql kv upsert
2020-09-02 10:12:44 -07:00
Chris Lu
ed62f52452
fix sql insert with duplicated primary key
2020-09-02 10:09:49 -07:00
Chris Lu
9ea290aa12
fix error message
2020-09-01 23:21:41 -07:00
Chris Lu
06a1188267
mysql/postgres: support kv operations
2020-09-01 22:48:36 -07:00
Chris Lu
aa40295f06
leveldb: report not found entry
2020-09-01 22:48:23 -07:00
Chris Lu
87d7312bf6
mysql/postgres: properly report entry not found
2020-09-01 22:47:57 -07:00
Chris Lu
72b0a5f1d1
mysql or postgres: log find error
2020-09-01 22:25:17 -07:00
Chris Lu
2815bbe6c0
go fmt
2020-09-01 21:59:26 -07:00
Chris Lu
37234bf3f8
filer store adds kv support
...
can compile now, need to implement those unimplemented
2020-09-01 21:59:00 -07:00
James Hartig
8e54e34576
volume: Don't unmount before deleting volume in copy
...
If we unmount first and then delete, the delete fails because the volume
was unmounted. Delete ends up doing the same thing as the unmount anyways.
2020-09-01 22:00:07 -04:00
Chris Lu
2b14ae5819
fix tests
2020-09-01 01:33:43 -07:00
Chris Lu
d91ec535b3
fix tests
2020-09-01 01:29:13 -07:00
Chris Lu
eb7929a971
rename filer2 to filer
2020-09-01 00:21:19 -07:00
Chris Lu
38e06d783d
volume: check disk space before compaction
...
fix https://github.com/chrislusf/seaweedfs/issues/1440
2020-08-31 18:10:53 -07:00
Chris Lu
dbb10e0f0b
testing with more than 30days
2020-08-31 17:59:29 -07:00
Chris Lu
dd176cdb8b
avoid tiny step pagination
2020-08-31 11:28:03 -07:00
Chris Lu
97733c3dff
typo
2020-08-31 10:49:17 -07:00
Chris Lu
c8398bdb09
adjust metrics
2020-08-31 10:41:05 -07:00
Chris Lu
8a1c8e41b3
simplify if else logic
2020-08-31 10:39:24 -07:00
Chris Lu
f0c89cfacd
go fmt
2020-08-31 10:23:31 -07:00
Chris Lu
edb9d65e05
Merge pull request #1431 from kmlebedev/wip-prefix-search
...
Wip prefix search
2020-08-31 10:22:14 -07:00
Konstantin Lebedev
9a195bebfd
accurate limit
2020-08-31 22:13:13 +05:00
Konstantin Lebedev
18b98f9747
not convert err to string
2020-08-31 21:55:18 +05:00
Konstantin Lebedev
22f32e75c4
rename
2020-08-31 21:52:05 +05:00
Konstantin Lebedev
a8de332beb
fix
2020-08-31 21:45:44 +05:00
Konstantin Lebedev
7f013d77d6
add var UnsupportedListDirectoryPrefixedErr
2020-08-31 21:43:32 +05:00
Konstantin Lebedev
60a86cfe04
add const
2020-08-31 21:35:16 +05:00
Konstantin Lebedev
82ea121d09
rm func
2020-08-31 18:50:06 +05:00
Konstantin Lebedev
2ea638f865
filer store wrapper can implement the logic to filter by prefi
2020-08-31 17:13:56 +05:00
Chris Lu
408e339c53
also delete the manifest chunk itself
2020-08-31 00:16:03 -07:00
Chris Lu
dd52037c27
1.92
2020-08-30 22:56:35 -07:00
Chris Lu
33d8c6c617
change log to warning
2020-08-30 21:01:44 -07:00
Chris Lu
e62d7f221f
a bit more logs
2020-08-30 21:00:24 -07:00
Chris Lu
22fe4ae573
mount: fix for UrBackup
2020-08-30 21:00:09 -07:00
Chris Lu
ae84a9ee2e
add logs
2020-08-30 20:59:04 -07:00
Chris Lu
a41588279a
change log level 5 to 4
2020-08-30 20:12:04 -07:00
Chris Lu
56244fb9a1
fix hard coded host address
2020-08-30 10:23:35 -07:00
Chris Lu
f2a8574448
filer and mount deletion resolves manifest chunks also
2020-08-30 02:07:14 -07:00
Chris Lu
99ecf63276
go fmt
2020-08-29 22:28:33 -07:00
Chris Lu
ef75ce8a34
use UTC for all time related folders
2020-08-29 21:01:14 -07:00
Chris Lu
b69cb74c03
read meta logs by timestamp
...
pass in event ts when moving logs
meta aggregator reads in memory logs only
2020-08-29 17:37:19 -07:00
Chris Lu
063c9ddac5
adjust logs
2020-08-29 11:56:22 -07:00
Chris Lu
ca658a97c5
add signatures to messages to avoid double processing
2020-08-28 23:48:48 -07:00
Chris Lu
63ad1abcce
watch: follow changes for a pattern
2020-08-28 19:43:04 -07:00
Chris Lu
b7a654a318
file mime avoid saving application/octet-stream
2020-08-28 19:42:40 -07:00
Chris Lu
707192f966
fix help message
2020-08-27 00:13:38 -07:00
Chris Lu
0be6863c87
rename
2020-08-26 22:40:15 -07:00
Chris Lu
50f7e2e89f
add log messages
2020-08-26 09:51:20 -07:00
Chris Lu
6bc618051c
log message
2020-08-26 09:32:56 -07:00
Chris Lu
9dae4e9d3e
fix logs
2020-08-26 09:31:11 -07:00
Chris Lu
4c1a3187bf
fix log error
2020-08-26 09:25:54 -07:00
Chris Lu
ab759f0ec2
erasure coding: fix EC error if multiple disks are configured in one volume server
2020-08-26 09:16:58 -07:00
Chris Lu
d1cf39f180
fix logging
2020-08-26 08:55:15 -07:00
Chris Lu
83012f543a
s: avoid possible completeMultipartUpload re-tries that can overwrite the uploaded file
2020-08-26 08:50:24 -07:00
Konstantin Lebedev
464d4c82ec
stop send heartbeat before stop volume server
2020-08-26 16:16:11 +05:00
Chris Lu
1901f15cd2
volume server: remove whitelist for status checking
...
Volume liveness/readiness not work if jwt enabled, I try use param
-whiteList
but jwt breaks
2020-08-25 02:27:47 -07:00
Chris Lu
81e5124faf
s3: list bucket permission change from admin to read
...
fix https://github.com/chrislusf/seaweedfs/issues/1430
2020-08-24 11:22:45 -07:00
Chris Lu
5f55a87101
close http response
2020-08-24 00:32:44 -07:00
Chris Lu
0b301b0b47
consume all response body in order to release requests
2020-08-23 21:32:29 -07:00
Chris Lu
2cdad77d82
fix checkin error
2020-08-23 18:30:11 -07:00
Chris Lu
81a8aa6581
1.91
2020-08-23 17:28:40 -07:00
Chris Lu
51346a5930
always keep the manifest list of chunks
2020-08-23 17:15:12 -07:00
Chris Lu
c1d1677a28
keep manifest chunks forever
2020-08-23 16:59:01 -07:00
Chris Lu
98175548c2
adjust deletion ordering
2020-08-23 16:27:12 -07:00
Chris Lu
f0e50e7b26
fix test
2020-08-23 16:12:34 -07:00
Chris Lu
aee27ccbe1
multiple fixes
...
* adjust isOpen count
* move ContinuousDirtyPages lock to filehandle
* fix problem with MergeIntoVisibles, avoid reusing slices
* let filer delete the garbage
2020-08-23 15:48:02 -07:00
Chris Lu
c19245886c
simpler test
2020-08-23 14:20:27 -07:00
Chris Lu
df816a58fe
add tests
2020-08-23 14:09:25 -07:00
Chris Lu
77393d3d30
add file handle locking when changing file entry
2020-08-23 00:35:50 -07:00
Chris Lu
d60bcbf08a
sorting chunks
2020-08-23 00:00:36 -07:00
Chris Lu
5e6b714836
add random test
2020-08-22 16:33:00 -07:00
Chris Lu
42ddbcc3a2
SQL insert falls back to update
...
fix https://github.com/chrislusf/seaweedfs/issues/1429
2020-08-20 07:52:46 -07:00
Chris Lu
f48567c5c6
remove unused function
2020-08-19 22:53:49 -07:00
Chris Lu
c45ba5d7d4
fix listObjectsV2 response format
...
fix https://github.com/chrislusf/seaweedfs/issues/1426 issue 1
2020-08-19 12:08:04 -07:00
James Hartig
3ccfa4c6ad
Added VolumeMarkWritable and VolumeStatus grpc methods
...
This is necessary for copy to mark as read-only and then restore the
original state afterwards.
2020-08-19 11:42:56 -04:00
Chris Lu
3b4b1d4a77
fix tests
2020-08-19 01:37:56 -07:00
Chris Lu
839634097f
also do flush on release
2020-08-19 01:27:10 -07:00
Chris Lu
c27e18aa6a
read possible old deleted chunks
2020-08-19 00:43:07 -07:00
Chris Lu
a78772d5ea
avoid shutdown in the middle of running
2020-08-19 00:42:02 -07:00
Chris Lu
ae9bc4a508
logs
2020-08-18 23:42:26 -07:00
Chris Lu
ed4b43b419
adjust logs
2020-08-18 23:42:09 -07:00
Chris Lu
9f1e0aeef5
delete chunks in the last step
2020-08-18 23:40:53 -07:00
Chris Lu
99d05f758c
adjust logs
2020-08-18 23:39:18 -07:00
Chris Lu
fe01191b5b
support read option readDeleted=true
2020-08-18 19:22:16 -07:00
Chris Lu
6ccd7f0a4d
refactoring
2020-08-18 18:01:37 -07:00
Chris Lu
c026eb0592
refactoring
2020-08-18 17:39:29 -07:00
Chris Lu
7e91ae592c
pass in option to read deleted entries
...
not working yet
2020-08-18 17:37:26 -07:00
Chris Lu
51ecb49db3
for debugging
2020-08-18 17:35:39 -07:00
Chris Lu
ee11d98650
refactoring
2020-08-18 17:35:19 -07:00
Chris Lu
6a92f0bc7a
refactoring to typed Size
...
Go is amazing with refactoring!
2020-08-18 17:04:28 -07:00
Chris Lu
618b2f6829
release resources only when needed to
2020-08-18 12:53:08 -07:00
Chris Lu
208849702d
logs
2020-08-18 12:52:54 -07:00
Chris Lu
1fcd083db3
printout data size
2020-08-18 09:09:29 -07:00
Chris Lu
3e5339337a
minor
2020-08-18 08:50:14 -07:00
Chris Lu
85001cbec7
properly report io.EOF
2020-08-18 08:18:54 -07:00
Chris Lu
0ca45a5cbc
adjust logs
2020-08-18 00:34:15 -07:00
Chris Lu
cd43738245
fix reading when filling zeros
2020-08-18 00:32:01 -07:00
Chris Lu
ecb3ce46be
adjust error logs
2020-08-18 00:31:42 -07:00
Chris Lu
30fe424469
add one more test case
2020-08-17 22:47:27 -07:00
Chris Lu
1b68ba953b
fix for out of range reads
2020-08-17 22:46:32 -07:00
Chris Lu
56fbd2c211
fix reading
2020-08-17 21:17:32 -07:00
Chris Lu
09e126bae5
refactoring: use interface
2020-08-17 20:20:08 -07:00
Chris Lu
be4d42b8e2
rename
2020-08-17 20:15:53 -07:00
Chris Lu
0625e63648
count 0 as part of the reads
2020-08-17 20:14:40 -07:00
Chris Lu
0be4b6e7f0
logs
2020-08-17 16:05:40 -07:00
Chris Lu
97e54a80d4
rename variables
2020-08-17 16:05:13 -07:00
Chris Lu
abdaf9958d
possibly read more
2020-08-17 16:04:56 -07:00
Chris Lu
9d46c7bc78
rename
2020-08-17 11:12:10 -07:00
Chris Lu
4ccfdaeb4d
prevent nil
2020-08-17 10:07:34 -07:00
Chris Lu
24c8e6bcb4
minor optimization
2020-08-17 10:03:34 -07:00
Chris Lu
f5837b7000
report error first
2020-08-16 23:49:10 -07:00
Chris Lu
2ac27616bc
fix possible out of range bytes
...
avoid buff out of range resp.Data = buff[:totalRead]
2020-08-16 23:47:34 -07:00
Chris Lu
ff200398bb
1.90
2020-08-16 21:18:50 -07:00
Chris Lu
4ceeba9e70
streaming reads
2020-08-16 21:07:46 -07:00
Chris Lu
e72953dff7
logs
2020-08-16 21:06:03 -07:00
Chris Lu
627b081b67
adjust logs
2020-08-16 16:32:22 -07:00
Chris Lu
22e5132b3a
adjust log level
2020-08-16 16:25:11 -07:00
Chris Lu
ee0f92a6be
reduce memory allocation
2020-08-16 16:24:40 -07:00
Chris Lu
2ba817afac
read randomly written data
2020-08-16 15:16:46 -07:00
Chris Lu
6111b265e7
fix compilation
2020-08-16 01:38:16 -07:00
Chris Lu
4a77f0820a
clean up logs
2020-08-16 01:37:50 -07:00
Chris Lu
8c9e6eaacd
fix tests
2020-08-16 01:35:52 -07:00
Chris Lu
b71df82292
1.89
2020-08-16 00:57:28 -07:00
Chris Lu
1d9ea30b72
fix ViewFromVisibleIntervals
2020-08-16 00:49:26 -07:00
Chris Lu
aec7f32b02
fix reader_at
2020-08-16 00:49:08 -07:00
Chris Lu
5d80fc2ec7
adjust logs
2020-08-15 21:09:31 -07:00
Chris Lu
e0bfd3161a
update metadata only if changed
2020-08-15 19:56:24 -07:00
Chris Lu
003d48da21
adjust logs
2020-08-15 19:55:28 -07:00
Chris Lu
6ee8d952d2
adjust log level
2020-08-15 18:24:35 -07:00
Chris Lu
3e1395b767
adjust log message
2020-08-15 17:06:16 -07:00
Chris Lu
a22ee30596
fix nil
2020-08-15 17:01:42 -07:00
Chris Lu
0d60e67816
ensure meta data changes are updated
2020-08-15 14:15:07 -07:00
Chris Lu
5b43bddf20
proper deletion ordering
...
delete central file store first, then delete local cache
2020-08-15 09:33:41 -07:00
Chris Lu
c647deace1
file size support set file length
...
use Attr.FileSize and TotalChunkSize to determine file size
2020-08-15 09:32:47 -07:00
Chris Lu
c03bb180eb
fix error reporting
2020-08-14 00:44:02 -07:00
Chris Lu
eb493283dd
deletion error report
2020-08-14 00:23:01 -07:00
Chris Lu
a7f669044e
rename also applies to open file handle
2020-08-14 00:22:49 -07:00
Chris Lu
edfa73782f
adjust log level
2020-08-14 00:22:21 -07:00
Chris Lu
0983060a90
increase default volume file size limit to 1024
...
avoid possible large chunk size set on mount or filer
2020-08-13 09:07:22 -07:00
Chris Lu
090612492c
fix compilation
2020-08-13 00:07:56 -07:00
Chris Lu
d43129d27f
add back handles lock
2020-08-12 23:52:13 -07:00
Chris Lu
f735d579d3
adjust comment
2020-08-12 13:22:19 -07:00
Chris Lu
8824a9755c
remove directory cache
2020-08-12 13:11:04 -07:00
Chris Lu
83cad3da79
add retry file upload
2020-08-11 20:30:11 -07:00
Chris Lu
152a6cbc2b
minor adjustments
2020-08-10 20:42:27 -07:00
Chris Lu
f1b40567e3
Merge pull request #1419 from robin1900/master
...
fix bug: two same volumeId in different collections
2020-08-10 20:38:55 -07:00
cheng.li01
25fbff5d52
fix bug: two same volumeId in different collections
...
1, there will be two leader when master server startup in a few seconds
2, raft server will get a leader even there is only one master, so there is no need to do hard code to set the server to be leader
2020-08-10 16:37:47 +08:00
Chris Lu
e74dc4e4bc
add back fs node cache for renaming
2020-08-09 21:56:09 -07:00
Chris Lu
f86c7d911a
remove println
2020-08-09 14:42:25 -07:00
Chris Lu
9ecc1170a3
istObjects和listObjectsV2不能查询子目录 #1418
...
fix https://github.com/chrislusf/seaweedfs/issues/1418
2020-08-09 14:35:53 -07:00
Chris Lu
3f4aff5dde
s3: fix delimiter in list response
2020-08-09 09:09:35 -07:00
Chris Lu
b056707770
1.88
2020-08-08 22:55:12 -07:00
Chris Lu
9832653e1d
FUSE mount: proper error with deleting non empty folder
2020-08-08 21:37:36 -07:00
Chris Lu
3b1a95ac26
filer refactoring: same auto chunking logic for POST and PUT, no size limit
2020-08-08 12:02:06 -07:00
Chris Lu
67348e7b15
less noisy heartbeat logs
2020-08-08 10:53:35 -07:00
Chris Lu
bee0d7e5eb
lower log priority for noisy heartbeat
2020-08-08 10:52:13 -07:00
Chris Lu
ae00cce4bd
support POST and PUT auto chunking
2020-08-08 10:45:37 -07:00
Chris Lu
bd8bfdae07
refactoring
2020-08-08 10:18:43 -07:00
Chris Lu
ab6e5c0dc4
adjust error message
2020-08-08 10:18:32 -07:00
Chris Lu
828a5ae429
check signature only when auth is enabled
2020-08-08 09:11:40 -07:00
Chris Lu
20e2ac1add
filer: store md5 metadata for files uploaded by filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1412
2020-08-06 10:04:17 -07:00
Chris Lu
93ea0801ea
volume: the variable for the master node may be stale?
...
related to https://github.com/chrislusf/seaweedfs/issues/1414
2020-08-06 09:48:54 -07:00
Chris Lu
4ecfa9879d
volume: report Content-MD5 in response header
2020-08-06 05:22:53 -07:00
Chris Lu
41007ced77
remove logging
2020-08-06 04:26:29 -07:00
Chris Lu
2b74abf766
S3: configurable access for anonymous user
...
fix https://github.com/chrislusf/seaweedfs/issues/1413
2020-08-06 03:41:34 -07:00
Устюжанин Антон Александрович
9ca011e3ca
ListDirectoryPrefixedEntries
2020-08-06 00:38:59 +05:00
Устюжанин Антон Александрович
b231f7bdab
ListDirectoryPrefixedEntries
2020-08-06 00:37:42 +05:00
Устюжанин Антон Александрович
a457c308ad
test ListDirectoryPrefixedEntries
2020-08-06 00:24:31 +05:00
Устюжанин Антон Александрович
2d3b355fb6
test ListDirectoryPrefixedEntries
2020-08-05 23:56:06 +05:00
Устюжанин Антон Александрович
67f134ebd7
test ListDirectoryPrefixedEntries
2020-08-05 23:44:48 +05:00
Устюжанин Антон Александрович
dc9fc01254
test ListDirectoryPrefixedEntries
2020-08-05 23:38:00 +05:00
Устюжанин Антон Александрович
33a9e5e2d1
test ListDirectoryPrefixedEntries
2020-08-05 22:19:16 +05:00
Chris Lu
cbd80253e3
better needle id format
2020-08-04 22:09:07 -07:00
Chris Lu
1b3a80dd3d
non-fatal error
2020-08-01 13:46:52 -07:00
Chris Lu
bbbea8159c
http request use gzip if possible
2020-08-01 13:20:52 -07:00
Chris Lu
2f03481cb2
in case when content is not compressed
2020-08-01 13:08:10 -07:00
Chris Lu
49199c69d4
volume: avoid special logic for .gz files
2020-08-01 11:19:57 -07:00
Chris Lu
f89186a3cf
better logging
2020-08-01 11:19:57 -07:00
Chris Lu
49d6dcfbc6
dynamically change the log level for reading the disk status
2020-08-01 11:19:57 -07:00
limd
b41b7ea4d0
fix spark read s3 bug (sc.binaryFiles)
2020-08-01 01:08:30 +08:00
Chris Lu
a6b59d50f7
1.87
2020-07-30 13:52:04 -07:00
Chris Lu
eed525b717
FUSE mount: remove DirListCacheLimit
...
outdated parameter
2020-07-28 09:24:39 -07:00
Chris Lu
a566bfc6e1
s3: use bucket in the domain
...
fix https://github.com/chrislusf/seaweedfs/issues/1405
2020-07-28 08:47:27 -07:00
limd
92c32f9d46
PutObject: fix create folder bug
2020-07-28 00:58:42 +08:00
Chris Lu
54cd582653
refactor
2020-07-26 12:58:58 -07:00
Chris Lu
c50ec1599c
auth for presigned put object part
2020-07-26 12:09:50 -07:00
Chris Lu
ea0180de4e
auth for presigned put operation
2020-07-26 12:06:45 -07:00
Chris Lu
3faa0aa448
rename
2020-07-25 20:09:19 -07:00
Chris Lu
023a1efdf2
set filename in Content-Disposition header
2020-07-25 20:06:40 -07:00
popstk
a09fcfcf20
fix s3api auth bug
2020-07-25 17:14:49 +08:00
popstk
feb52400be
fix s3api copy object handler SerializationError
2020-07-25 15:52:31 +08:00
Chris Lu
d40de39e75
FUSE: do not change crtime, uid, gid on save
2020-07-24 10:13:45 -07:00
Chris Lu
8dfeba8023
FUSE: skip changing to empty uid and gid during flush
...
Skip uid and gid changes if request uid and gid are zero
mongodb lock file, and many interim files, has empty uid and gid
I0724 10:05:51 93643 filehandle.go:173] /db/diagnostic.data/metrics.interim fh 1333342842031408359 flush Flush [ID=0x3 Node=0x50 Uid=0 Gid=0 Pid=178] 0x10 fl=0x0 lk=0x0
2020-07-24 10:06:43 -07:00
Chris Lu
dd29b8c81d
fix compilation
2020-07-24 09:36:05 -07:00
Chris Lu
5788f1f6bf
remove fsnode cache
...
fix https://github.com/chrislusf/seaweedfs/issues/1400
2020-07-24 09:25:54 -07:00
Chris Lu
6962ca48d3
fix compilation
2020-07-23 21:31:23 -07:00
Chris Lu
df7eee523b
FUSE: implement dir fsync
...
fix https://github.com/chrislusf/seaweedfs/issues/1398
2020-07-23 21:09:40 -07:00
Chris Lu
3bbac0820b
FUSE: faster rename implementation
2020-07-23 21:08:42 -07:00
Chris Lu
943fa4ea0c
fix compilation error
2020-07-22 16:11:17 -07:00
Chris Lu
49446af516
simplify
2020-07-22 13:43:54 -07:00
James Hartig
229f11c660
Added VolumeNeedleStatus volume server grpc method
...
This is needed for the diffing tool to get the cookie for a needle
2020-07-22 15:02:21 -04:00
James Hartig
0be3bb4a87
idx: Switch WalkIndexFile to accept generic io.ReaderAt
...
This is necessary to use it with in-memory idx files.
2020-07-22 15:01:16 -04:00
Chris Lu
7a7e70b6cc
1.86
2020-07-20 23:39:58 -07:00
Chris Lu
885c624bce
volume.fsck: follow manifest chunks
2020-07-20 22:02:05 -07:00
Chris Lu
44057a4de1
clean up chunks in manifest
2020-07-20 22:01:39 -07:00
Chris Lu
0b2e06268b
use merge factor 1000
2020-07-20 18:30:09 -07:00
Chris Lu
d02c0fe0c0
refactoring
2020-07-20 03:34:06 -07:00
Chris Lu
60d14a9800
mount: fix difference with storage format in local cache
2020-07-20 00:17:50 -07:00
Chris Lu
97d97f3528
go code can read and write chunk manifest
2020-07-19 17:59:43 -07:00
李远军
4162766c46
Fix the bug of " http://xx/xx,xxx.PNG?width=100 "
...
If you want to do a resize a UPPER ext pic file, there is a bug caused by commit 308688c8d0
(also see https://github.com/chrislusf/seaweedfs/blob/master/weed/images/resizing.go#L43 )
2020-07-17 19:57:35 +08:00
Chris Lu
f43146b237
resolve directories if containing home directory
2020-07-16 22:50:14 -07:00
Chris Lu
befb396892
mount: resolve home directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1391
2020-07-16 22:38:55 -07:00
Chris Lu
2b2ca7d1dc
stop if the directory can not be found
...
related to https://github.com/chrislusf/seaweedfs/issues/1391
2020-07-16 13:24:53 -07:00
Chris Lu
2286d27730
rename
2020-07-15 14:19:15 -07:00
Chris Lu
0db4204c81
add is_file_chunks to filer.proto
2020-07-14 22:42:52 -07:00
Chris Lu
bbc8ae110f
report errors
2020-07-14 11:27:00 -07:00
Chris Lu
17f2d05cde
filer: ensure logs are persisted
2020-07-14 11:25:50 -07:00
Chris Lu
ddec7b2bb9
go fmt
2020-07-14 07:34:16 -07:00
Chris Lu
4f6096c7f0
add reading from persisted logs for local filer store
2020-07-13 22:55:28 -07:00
Chris Lu
87b5031714
change variable visibility
2020-07-13 08:19:48 -07:00
Chris Lu
7597831cac
filer: leveldb2 supports peers also
2020-07-13 00:05:40 -07:00
Chris Lu
5ba894bb72
leveldb2: support subscribing from peers
2020-07-13 00:05:40 -07:00
Chris Lu
5a4f258a8c
1.85
2020-07-12 17:34:09 -07:00
Chris Lu
4eecc6abc6
filer: emit metadata notification correctly for batch deletion
2020-07-12 17:32:19 -07:00
Chris Lu
1dd2c76532
refactoring
2020-07-12 17:31:24 -07:00
Chris Lu
224103e13b
aggregated logs has empty flushFn
2020-07-11 09:12:03 -07:00
Chris Lu
bfabb4159b
refactoring
2020-07-11 09:11:15 -07:00
Chris Lu
f2e8ad6241
refactoring
2020-07-11 06:35:54 -07:00
Chris Lu
2d1df97933
mount: fatal error if can not process meta data
2020-07-11 06:16:48 -07:00
Chris Lu
e3e00d36c5
reset cache for each new release
2020-07-11 06:16:17 -07:00
Chris Lu
7ad0bce2f5
remove old code
2020-07-10 23:03:22 -07:00
Chris Lu
e4ef3814e7
fix compilation
2020-07-10 10:02:59 -07:00
Chris Lu
e7c04af1d0
skip http.StatusPartialContent in case of error and superfluous response.WriteHeader
2020-07-10 08:56:23 -07:00
李远军
8c318470dd
Revert "remove fixJpgOrientation"
...
This reverts commit de5ca9b2
minimise the revert changes
2020-07-10 10:28:28 +08:00
李远军
6608cb5f43
Revert "remove fixJpgOrientation"
...
This reverts commit de5ca9b2
2020-07-10 10:08:36 +08:00
cheng.li01
dad1161c70
fix dn.volumes Iterate when write issue
2020-07-08 19:57:19 +08:00
Chris Lu
54d6b3c30f
filer: support running multiple mount to different filers avoiding duplicated cache
...
fix https://github.com/chrislusf/seaweedfs/issues/1382
2020-07-07 23:18:44 -07:00
Chris Lu
4bed1663e2
adjust log level
2020-07-07 23:17:31 -07:00
Chris Lu
482d13468b
remove println
2020-07-07 23:17:17 -07:00
Chris Lu
a2eb680f34
filer: default filer store directory
...
1. set default filer store directory
2. set peers, avoiding empty string counted as 1.
2020-07-07 23:06:48 -07:00
Chris Lu
3c269da37f
fix compilation
2020-07-06 09:46:24 -07:00
Chris Lu
d198e8c6d6
filer: support cross filer meta data sync if sharing the same store
2020-07-06 09:24:54 -07:00
Chris Lu
0b8cc9b626
aggregate multiple filer metadata chagne events
2020-07-05 23:05:02 -07:00
Chris Lu
49929e0869
rename
2020-07-05 15:52:36 -07:00
Chris Lu
70d8a3a1d3
add SubscribeLocalMetadata without checking persisted meta logs
2020-07-05 15:50:07 -07:00
Chris Lu
55e40b08fc
refactoring
2020-07-05 15:43:06 -07:00
Chris Lu
881e0fde2e
rename
2020-07-05 08:55:09 -07:00
Chris Lu
353bea8ddb
refactoring
2020-07-03 22:25:35 -07:00
Chris Lu
ce3630e7a4
refactoring
2020-07-03 22:16:59 -07:00
Chris Lu
59152334c7
default to 1%
2020-07-03 16:42:53 -07:00
Chris Lu
acf4a9ea99
volume: min free space default to 1%
2020-07-03 16:42:02 -07:00
Chris Lu
4d1484628a
refactoring
2020-07-03 16:41:30 -07:00
Chris Lu
faa5c2e89a
refactoring
2020-07-03 16:34:31 -07:00
Chris Lu
37d5b3ba12
replication: pass isFromOtherCluster also to EventNotification
...
EventNotification is consistent with message queue and metadata logs.
2020-07-01 08:06:20 -07:00
Chris Lu
31e23e9783
filer: support active<=>active filer replication
2020-06-30 22:53:57 -07:00
Chris Lu
df75b5c98d
passing ctx for metadata logging
2020-06-28 14:34:51 -07:00
Chris Lu
12f6b683fc
default to 8 volumes
...
in case all the volumes are allocated for a collection
2020-06-28 14:34:21 -07:00
Chris Lu
42a338d7b0
FUSE mount: clean up file handles during renaming
...
resolve dir rename when file is still open. Need to clean the file handles as soon as possible.
These can happen out of order:
file rename, then file release
file release, then file rename
2020-06-28 13:41:00 -07:00
Chris Lu
95a1860d65
remove empty calls for list cache
2020-06-28 10:25:54 -07:00
Chris Lu
6498d0c86b
remove list cache
2020-06-28 10:21:52 -07:00
Chris Lu
9033a7d369
removed async option
2020-06-28 10:18:32 -07:00
Chris Lu
1bb8cae65d
reverting and working
2020-06-28 10:14:17 -07:00
Chris Lu
b813fac4a3
1.84
2020-06-28 07:28:21 -07:00
Chris Lu
20d9cfae86
volume: following md5 base64 encoding
...
fix https://github.com/chrislusf/seaweedfs/issues/1374
2020-06-28 07:25:50 -07:00
Chris Lu
f0f884edd5
1.83
2020-06-27 15:46:59 -07:00
Chris Lu
c2797cc6de
add logs
2020-06-27 15:46:53 -07:00
Chris Lu
62260e526b
add logging
2020-06-27 12:54:37 -07:00
Chris Lu
bef356ce4c
since we already know the chunk size, no need to iterate
2020-06-27 12:51:04 -07:00
Chris Lu
a808b3b5df
incase the memory data is too small
2020-06-27 11:59:15 -07:00
Chris Lu
3dbd51c3c2
a little bit more efficient
2020-06-26 10:02:37 -07:00
Chris Lu
212b6e7d42
error logging
2020-06-26 10:01:55 -07:00
Chris Lu
48b23f2fdd
FUSE mount: prevent concurrent modification
2020-06-26 10:00:48 -07:00
Chris Lu
3cec4b3c49
FUSE mount: fix file id written twice?
...
fix https://github.com/chrislusf/seaweedfs/issues/1373
2020-06-25 20:11:49 -07:00
Chris Lu
212c64241b
checking ecx file
2020-06-25 16:36:06 -07:00
Chris Lu
a4dc25cd22
volume: generate ec shard first, before generating the ec index file.
...
possible fix https://github.com/chrislusf/seaweedfs/issues/1364
the ec index file is used to determine whether the ec files are ready or not
2020-06-25 11:06:58 -07:00
Chris Lu
9fd7cdadf1
fix
2020-06-25 10:45:34 -07:00
Chris Lu
3b638d3994
add more ec encoding logging
2020-06-25 09:43:38 -07:00
Chris Lu
e8e0d629af
ensure correct file mode
2020-06-24 23:26:54 -07:00
Chris Lu
32a1f22730
more logs
2020-06-24 22:07:53 -07:00
Chris Lu
5be12eea37
zstd fix
2020-06-24 11:39:12 -07:00
Chris Lu
2ff37ccdbd
testing parsing the upload
...
The mime type is always the value passed in.
Compress or not depends on the content detection, file name extension, and compression ratio.
If the content is already compressed, need to know the content size.
2020-06-24 11:39:12 -07:00
Chris Lu
c21f4ebfee
Merge pull request #1371 from Kimbsen/content_md5_validation
...
Optional md5 validation of uploads
2020-06-24 11:33:44 -07:00
Kimbsen
4eeab2a379
remove unused imports
2020-06-24 13:37:00 +02:00
Kimbsen
ffddecebef
do md5 validation AFTER decompression
2020-06-24 13:35:13 +02:00
Chris Lu
fe60db404a
prepare for zstd
2020-06-23 09:12:02 -07:00
Kimbsen
6b1e93ba0b
check request header not part header for content-encoding
2020-06-23 17:04:24 +02:00
Kimbsen
38626cb584
don't forget the last few bytes
2020-06-23 16:53:49 +02:00
Kimbsen
8d1f6cd567
added optional md5 verification of uploaded data
2020-06-23 16:24:11 +02:00
Chris Lu
a4e3cffe0a
S3: DeleteObject API also support deleting a folder recursively
2020-06-22 10:01:00 -07:00
Chris Lu
98493e0152
volume: add default minFreeSpacePercent
...
fix https://github.com/chrislusf/seaweedfs/issues/1366
2020-06-21 08:44:06 -07:00
Chris Lu
e7c36f740a
refactoring
2020-06-21 08:38:00 -07:00
Chris Lu
8b4a32e782
adjust help message
2020-06-21 08:34:24 -07:00
Chris Lu
b4ec04d464
fix the help message
2020-06-21 08:31:20 -07:00
Chris Lu
de5ca9b258
remove fixJpgOrientation
2020-06-20 12:50:40 -07:00
Chris Lu
3080c197e3
rename UnCompressData to DecompressData
2020-06-20 08:16:16 -07:00
Chris Lu
97239ce6f1
rename filechunk is_gzipped to is_compressed
2020-06-20 08:15:49 -07:00
Chris Lu
a735fd767c
adjust uncompress data
2020-06-20 08:01:00 -07:00
Chris Lu
ca3516ac6d
adjust protoc
2020-06-20 08:00:25 -07:00
Chris Lu
e912fd15e3
renaming
2020-06-19 22:45:27 -07:00
Chris Lu
16fe132a20
detect mime type before replicating to other volume servers
2020-06-19 22:11:36 -07:00
Chris Lu
044841c885
master: always clear previous master meta data directory
2020-06-19 20:42:16 -07:00
Chris Lu
2eeb7551f9
less noisy disk size checking
2020-06-19 15:06:33 -07:00
Chris Lu
c48b407053
1.82
2020-06-19 13:47:27 -07:00
Chris Lu
b78a2dc59b
adjust help message
2020-06-19 12:55:09 -07:00
Chris Lu
f7a45d448f
FUSE mount: lazy loading meta cache
2020-06-19 09:45:42 -07:00
Chris Lu
c0283eee1a
more test cases
2020-06-19 09:45:42 -07:00
Chris Lu
55b6efb755
fix checking visited nodes
2020-06-19 09:45:42 -07:00
Chris Lu
0e7c1a300b
Merge pull request #1363 from DXist/fix_utc
...
Return last modified in UTC
2020-06-19 08:49:00 -07:00
Rinat Shigapov
320d70d305
return last modified in UTC
2020-06-19 10:26:40 +03:00
Chris Lu
546709c1de
check has visited by the boundary
2020-06-18 22:28:17 -07:00
Chris Lu
0e3b970c0c
added bounded tree to track exploration boundary
2020-06-18 21:09:58 -07:00
Chris Lu
ae1994cbc1
erasure coding: fix cases where there are no .ecj files
2020-06-18 09:52:35 -07:00
Chris Lu
b74eced7a3
mount: a fix to prevent possible repeated calls
...
related to https://github.com/bazil/fuse/issues/130
2020-06-17 13:55:16 -07:00
Chris Lu
7ccc31853c
hard limit volume file size
2020-06-16 00:27:48 -07:00
Chris Lu
48880edc2c
volume: fix typo in minFreeSpacePercent option
2020-06-14 20:37:05 -07:00
Chris Lu
a0464fc267
fix help
...
fix https://github.com/chrislusf/seaweedfs/issues/1359
2020-06-12 22:06:21 -07:00
Chris Lu
0d1d83a5a5
mount: add option to auto create dir to mount
2020-06-12 11:46:58 -07:00
Chris Lu
00f96d86a1
fix typo
2020-06-11 10:53:25 -07:00
Chris Lu
63a1db64c5
Merge pull request #1358 from DXist/s3fixes
...
Return NotSuchKey error code for bucket S3 DeleteObject method
2020-06-11 10:37:19 -07:00
Rinat Shigapov
283b749ff1
keep 204 response
2020-06-11 19:00:34 +03:00
Rinat Shigapov
fafc41a27f
return xml encoded NotFound status code for s3 delete
2020-06-11 17:53:15 +03:00
Rinat Shigapov
2d2c5dfa39
proxy status code of filer delete response
2020-06-11 15:00:47 +03:00
Chris Lu
628b27ef3b
purge old cache implementation
2020-06-11 01:50:00 -07:00
Chris Lu
b9365de47b
fix volume server pprof
2020-06-10 13:10:10 -07:00
Chris Lu
718052656a
Merge pull request #1355 from Kimbsen/add_http_pprof
...
Add --pprof to volume servers to enable http pprof handlers
2020-06-10 13:05:04 -07:00
Chris Lu
95a9686eaf
1.81
2020-06-10 13:00:42 -07:00
Chris Lu
c66f357715
report error correctly
2020-06-10 10:59:25 -07:00
Kimbsen
03e8f976d9
added --pprof to volume servers
2020-06-10 11:52:24 +02:00
Chris Lu
c38066be89
also add it for webdav
2020-06-09 18:07:03 -07:00
Chris Lu
4598473251
fix possible EOF problem
2020-06-09 18:04:40 -07:00
Chris Lu
bb3e86fbd5
auto create non-existing cache directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1350
2020-06-08 08:49:57 -07:00
Chris Lu
bdbe5ddbe9
volume server: add a fix for dead lock
...
https://github.com/chrislusf/seaweedfs/issues/1353
2020-06-08 08:12:59 -07:00
Chris Lu
b8e346cc86
1.80
2020-06-05 15:29:25 -07:00
Chris Lu
45a9d852d3
go fmt
2020-06-05 15:27:10 -07:00
Chris Lu
03c2f03509
Merge pull request #1351 from ekozlov-search/master
...
Set volumes ReadOnly if low free disk space
2020-06-05 09:56:46 -07:00
Chris Lu
ebd1759741
fix ReadAt
...
fix https://github.com/chrislusf/seaweedfs/issues/1344
2020-06-05 09:54:14 -07:00
Evgenii Kozlov
2410ffbd79
Name fix
2020-06-05 19:00:38 +03:00
Evgenii Kozlov
0e0db70f55
Set volumes ReadOnly if low free disk space
2020-06-05 18:18:15 +03:00
Chris Lu
393ab07c7c
disable pulse seconds
...
there are a few other hard coded timing settings, which will cause disconnected volume servers.
2020-06-04 10:52:01 -07:00
Evgenii Kozlov
48f9ff52cf
Discs statistics on Windows platform.
2020-06-02 11:52:16 +03:00
Chris Lu
bc2ec6774d
inject git version into build
2020-06-02 00:10:38 -07:00
bingoohuang
1f8782a1ed
try showing the first 100 volume ids and an extra ...
2020-05-29 16:15:33 +08:00
bingoohuang
1a642b9876
add Volume Ids column only for max 100 volumes for convenience in the master ui.
2020-05-29 15:37:58 +08:00
bingoohuang
accb4964b7
util.BytesToHumanReadable
2020-05-29 10:00:07 +08:00
bingoohuang
21d0a013d5
rename 'needle' which collides with imported package name
2020-05-28 17:00:00 +08:00
bingoohuang
41cab5d69f
log JSON response if httpStatus >= 400
2020-05-28 08:56:55 +08:00
Chris Lu
ef2b3a0801
leveldb recover error handling
2020-05-26 00:03:44 -07:00
Chris Lu
5d3ec22975
refactoring
2020-05-26 00:03:31 -07:00
Chris Lu
d4235afe4d
support multiple locks
2020-05-25 18:39:53 -07:00
bingoohuang
ecdeef8c66
simplify func(w http.ResponseWriter, r *http.Request) to http.HandlerFunc
2020-05-25 21:00:12 +08:00
Chris Lu
e5a0787653
filer: leveldb store add auto recovery, report correct error
...
fix https://github.com/chrislusf/seaweedfs/issues/1326
2020-05-22 10:54:42 -07:00
Chris Lu
ed9e4dc749
1.79
...
fix compilation problem
2020-05-19 08:21:51 -07:00
Chris Lu
7ff2b0121d
specify constant data type to uint64 instead of default int
...
fix https://github.com/chrislusf/seaweedfs/issues/1324
2020-05-19 07:55:23 -07:00
Chris Lu
11b33c0760
fix compilation problem
2020-05-18 10:53:31 -07:00
Chris Lu
ac48c899e8
add printouts for synchronizing metadata which can be slow for large folders
2020-05-18 10:31:12 -07:00
Chris Lu
0c856e8387
avoid possible wrong arguments
...
fix https://github.com/chrislusf/seaweedfs/issues/1320
2020-05-18 10:07:12 -07:00
Chris Lu
2bfd810912
combine msgBroker into weed server command
2020-05-17 22:44:20 -07:00
Chris Lu
85aeead3d2
1.78
2020-05-17 20:28:04 -07:00
Chris Lu
8005786d9a
logs
2020-05-17 20:27:10 -07:00
Chris Lu
75179544c3
logging
2020-05-17 20:20:21 -07:00
Chris Lu
91b91d6cb7
add error to avoid copying not found volume
...
fix https://github.com/chrislusf/seaweedfs/issues/1317
2020-05-17 20:20:12 -07:00
Chris Lu
d3925d086c
add delete channel
2020-05-17 17:42:42 -07:00
Chris Lu
e0e31e67a8
Merge pull request #1318 from chrislusf/msg_channel
...
Add messaging, add channel
2020-05-17 17:39:16 -07:00
Chris Lu
081ee6fe34
reduc logs
2020-05-17 17:38:31 -07:00
Chris Lu
bff09fb74f
fix looping
2020-05-17 17:38:21 -07:00
Chris Lu
f5684839a2
add DeleteTopic
2020-05-17 17:34:10 -07:00
Chris Lu
3a57aef7a9
sync subscribe()
2020-05-17 17:33:53 -07:00
Chris Lu
95ca9dd8a2
subscribe support cancel
2020-05-17 11:10:45 -07:00
Chris Lu
f11233cd49
simplify disconnected stream detection
2020-05-17 08:57:47 -07:00
Chris Lu
1ea9bc66d9
avoid leaking grpc.NewClientStream
2020-05-17 08:56:33 -07:00
Chris Lu
85b53ac510
detect disconnected subscribers
2020-05-16 18:53:54 -07:00
Chris Lu
759cda0fe2
log
2020-05-16 18:47:41 -07:00
Chris Lu
b0de01ff3b
able to detect disconnected subscribers
2020-05-16 08:57:29 -07:00
Chris Lu
e02a8c67da
revert to one subscriber one thread
2020-05-15 21:38:42 -07:00
Chris Lu
b4e02ec525
fix prometheus problem
...
metrics.go:123] could not push metrics to prometheus push gateway 192.168.xx:9091: unexpected status code 200 while pushing to http://192.168.xx:9091/metrics/job/filer/instance/m111p112%3A8888 :
2020-05-14 21:08:34 -07:00
Chris Lu
3fbf635308
Merge branch 'master' into msg_channel
2020-05-13 03:54:15 -07:00
Chris Lu
3a5e4769cf
prevent empty locations returned
...
fix https://github.com/chrislusf/seaweedfs/issues/1313
2020-05-13 03:46:38 -07:00
Chris Lu
ca4017dd87
rename
2020-05-12 22:55:55 -07:00
Chris Lu
25257acd51
rename
2020-05-12 21:26:49 -07:00
Chris Lu
a7959c1c48
multiple subscriber with same subscriberId shares the topic manager
...
rename topicControl to topicCursor
2020-05-12 21:26:02 -07:00
Chris Lu
2f243f5b0b
refactor
2020-05-12 08:48:00 -07:00
Chris Lu
4b7fa31468
ensure montonically increasing tsNs
2020-05-11 01:53:54 -07:00
Chris Lu
d5a8297a1c
able to connect publisher
2020-05-10 23:45:48 -07:00
Chris Lu
d58371c5cd
Merge branch 'master' into msg_channel
2020-05-10 23:44:59 -07:00
Chris Lu
015dd3a147
batch file id deletion
2020-05-10 23:25:39 -07:00
Chris Lu
39e72fb23c
go fmt
2020-05-10 03:50:30 -07:00
Chris Lu
6bf3eb69cb
async chan write read, no write for closed chan
2020-05-10 03:48:35 -07:00
Chris Lu
78afb8bf46
Merge branch 'master' into msg_channel
2020-05-09 00:43:58 -07:00
Chris Lu
d693e77418
add pub sub md5
2020-05-09 00:43:53 -07:00
Chris Lu
07d7abe428
add deleteTopic, refactoring
2020-05-09 00:31:34 -07:00
Chris Lu
dfccc3c263
able to read chan and write chan
2020-05-08 02:47:22 -07:00
zhangsong
b161baa3f0
fix volume vacuum test unit
2020-05-07 19:58:06 +08:00
Chris Lu
a8bc8eb351
Merge branch 'master' into msg_channel
2020-05-07 03:18:24 -07:00
Chris Lu
73087f59b1
filer store: mongodb deleting children
...
fix https://github.com/chrislusf/seaweedfs/issues/1303
2020-05-07 03:11:26 -07:00
Chris Lu
be1d9f324f
Merge branch 'master' into msg_channel
2020-05-07 01:43:34 -07:00
Chris Lu
0952eb2854
Merge pull request #1302 from song-zhang/master
...
write request keep compatible with before
2020-05-06 15:37:34 -07:00
Chris Lu
5568395edd
Revert "Revert "Merge pull request #1299 from song-zhang/master""
...
This reverts commit afb6a1dbb4
.
2020-05-06 15:37:17 -07:00
zhangsong
3703c7e176
write request keep compatible with before
2020-05-06 21:35:32 +08:00
Chris Lu
1e3e4b3072
add broker connects to filer
2020-05-05 02:05:28 -07:00
Chris Lu
afb6a1dbb4
Revert "Merge pull request #1299 from song-zhang/master"
...
This reverts commit 9016fa19ba
, reversing
changes made to 47234760f4
.
2020-05-04 20:34:26 -07:00
Chris Lu
9016fa19ba
Merge pull request #1299 from song-zhang/master
...
use async write to persistent file to disk - part1
2020-05-04 15:51:03 -07:00
zhangsong
f9e8702bb4
use async write to persistent file to disk - part1
2020-05-04 17:39:44 +08:00
Chris Lu
47234760f4
log adjust format
2020-05-02 01:33:49 -07:00
Chris Lu
97a504b0bb
fix possible directory deletion error
2020-05-02 01:11:53 -07:00
Chris Lu
78422c2f69
Merge pull request #1298 from song-zhang/master
...
add fsync option for benchmark
2020-04-30 20:16:42 -07:00
Chris Lu
871efa4fc1
refactoring
...
some previous chunk etag was using md5, which should be wrong.
2020-04-30 17:20:47 -07:00
zhangsong
d819865fe7
add fsync option for benchmark
2020-04-30 23:31:08 +08:00
Chris Lu
ea809a45ab
add cipher key
2020-04-30 03:05:43 -07:00
Chris Lu
f9b6178b8f
log messages
2020-04-30 03:05:34 -07:00
Chris Lu
8c73410a51
subscribe from a timestamp
2020-04-30 02:19:51 -07:00
Chris Lu
4e16a90454
for debugging
2020-04-30 02:19:32 -07:00
Chris Lu
a25a8d8822
s3: print out time in UTC format
...
fix https://github.com/chrislusf/seaweedfs/issues/1297
2020-04-29 21:35:24 -07:00
Chris Lu
e93588ec78
FUSE mount: atomic local cache updates
2020-04-29 18:20:54 -07:00
Chris Lu
9e72e9e4b8
able to subscribe any topic from any point of time
2020-04-29 17:40:08 -07:00
Chris Lu
ed3cf811f5
refactoring
2020-04-29 13:26:02 -07:00
Chris Lu
7c10602b49
read in case cross chunks
2020-04-29 02:42:58 -07:00
Chris Lu
f9da859720
better error message
2020-04-29 02:42:06 -07:00
Chris Lu
d57559d3f7
chunk rememober encryption or not
2020-04-29 02:41:47 -07:00
Chris Lu
5a0986dca0
reset on getting the buffer
2020-04-29 02:41:01 -07:00
Chris Lu
8d42a1263b
meta subscription: update the last read time
2020-04-28 17:30:04 -07:00
Chris Lu
31583b5d6c
master and volume server: avoid race condition
...
The volume server may disconnect and reconnect to the same master.
The master's unregistration may happen after the reconnection. Thus the volume server will disappear.
2020-04-28 17:29:10 -07:00
Chris Lu
98edec7c3e
remove printouts
2020-04-28 16:45:48 -07:00
Chris Lu
8ed490164e
refactoring
2020-04-28 02:05:44 -07:00
Chris Lu
66effaed9e
fix compilation
2020-04-28 02:05:32 -07:00
Chris Lu
eab4c9219c
randomize benchmark content
...
fix https://github.com/chrislusf/seaweedfs/issues/1294
2020-04-28 01:50:56 -07:00
Chris Lu
fb81f12686
filer: simplify image resize
2020-04-28 00:05:57 -07:00
Chris Lu
5c57297bd1
metadata log: read from any timestamp
2020-04-28 00:05:57 -07:00
wuyuxiang
6850d28d6b
refacotr: move signal handling and pprof to grace package
2020-04-28 14:10:23 +08:00
Chris Lu
2a7957b4ca
FUSE: subcribe meta events based on timestamp
2020-04-27 16:52:20 -07:00
Chris Lu
6190fd665d
printout error
2020-04-27 12:41:31 -07:00
Chris Lu
725e47dcc3
filer: pass along user name and group name for HDFS
2020-04-27 05:48:48 -07:00
Chris Lu
508ec01ab0
tweaking
2020-04-26 16:11:06 -07:00
Chris Lu
f4fcde05b6
1.77
2020-04-26 05:54:47 -07:00
Chris Lu
b52b8ec685
Hadoop: fix entry not found for HCFS
...
also fix cipher related changes.
2020-04-26 05:21:54 -07:00
Chris Lu
0c2248f83a
add lock/unlock to example admin scripts
2020-04-23 18:41:48 -07:00
Chris Lu
913aa0f14e
mount: default to enable asyncMetaDataCaching
2020-04-23 14:14:28 -07:00
Chris Lu
662b5d0cf7
support semicolon seperated command lines
2020-04-23 14:01:46 -07:00
Chris Lu
73564e6a01
master: add cluster wide lock/unlock operation in weed shell
...
fix https://github.com/chrislusf/seaweedfs/issues/1286
2020-04-23 13:37:31 -07:00
Chris Lu
bdc337a719
make lock/unlock optional
2020-04-23 03:50:05 -07:00
Chris Lu
57ec7f2ba9
fix builds
2020-04-23 03:43:45 -07:00
Chris Lu
eab9701823
allow lock with an existing lock
2020-04-23 03:35:52 -07:00
Chris Lu
842e3301ee
make second lock instant
2020-04-23 03:32:18 -07:00
Chris Lu
69f336e59f
shell: add lock/unlock command
2020-04-23 03:11:07 -07:00
Chris Lu
ff0a7c1d18
clean up
2020-04-23 03:10:47 -07:00
Chris Lu
30ee4f3291
add exclusive lock library on shell
2020-04-23 02:31:04 -07:00
Chris Lu
77873b832b
add master side code for cluster wise exclusive lock
2020-04-23 01:55:44 -07:00
Chris Lu
369aa8a10a
1.76
2020-04-22 23:31:59 -07:00
Chris Lu
69eda558ef
reduce the connections
...
fix https://github.com/chrislusf/seaweedfs/issues/1277
this avoids sending error to wdclients, and clients retry the connections
2020-04-22 20:43:34 -07:00
Chris Lu
aebe39a803
avoid repeated grpc connection creation
...
fix https://github.com/chrislusf/seaweedfs/issues/1277
2020-04-22 18:26:24 -07:00
Chris Lu
5d0e1d8d74
also writes to local meta cache
...
before waiting for subscribed meta events
2020-04-22 15:40:47 -07:00
Chris Lu
72c5dba4d4
1.75
2020-04-21 22:03:07 -07:00
Chris Lu
a207285af7
cache metadata on startup
2020-04-21 22:00:34 -07:00
Chris Lu
9f9826e95a
refactor
2020-04-21 21:28:47 -07:00
Chris Lu
e24b25de78
async meta caching: can stream updates now
2020-04-21 21:16:13 -07:00
Chris Lu
4f02f7121d
read from meta cache
...
meta cache is not initialized
2020-04-21 18:50:30 -07:00
Chris Lu
b8e4238ad2
add placeholder for cached meta data reading
2020-04-21 18:02:08 -07:00
Chris Lu
27128c7875
adjust logging
2020-04-21 16:13:13 -07:00
Chris Lu
bafa95045b
volume: deletion checks all disk locations
...
fix https://github.com/chrislusf/seaweedfs/issues/1283
2020-04-21 14:49:58 -07:00
Chris Lu
57df14f76f
report error if copying failed
2020-04-21 14:28:46 -07:00
Chris Lu
cf54d70311
filer: add -ip.bind option
2020-04-21 14:21:06 -07:00
Chris Lu
5c348087dc
messaging: able to pub sub multiple partitions
2020-04-21 00:59:55 -07:00
Chris Lu
cb3985be70
go fmt
2020-04-20 17:48:06 -07:00
Chris Lu
5eb83dfbd3
add memory buffer size limit
2020-04-20 17:43:50 -07:00
Chris Lu
258fba8a0f
continue for reading from sealed memory buffer
2020-04-20 17:28:18 -07:00
Chris Lu
4bf959edf0
message broker: read also from sealed memory buffer
2020-04-20 17:26:38 -07:00
Chris Lu
bd43c62fbd
lower the test criteria
2020-04-20 03:08:10 -07:00
Chris Lu
c3f9fdf48f
adjust test
2020-04-20 02:54:21 -07:00
Chris Lu
5e8bb86995
close the connection with better accouting
2020-04-20 02:53:50 -07:00
Chris Lu
978da73cdb
revert this grpc related change
...
maybe related to https://github.com/chrislusf/seaweedfs/issues/1277
2020-04-20 02:53:22 -07:00
Chris Lu
ebfab42a50
refactoring
2020-04-20 00:08:47 -07:00
Chris Lu
2955b96ef1
refactor
2020-04-19 23:54:32 -07:00
Chris Lu
df25ed3c5b
logs
2020-04-19 23:37:50 -07:00
Chris Lu
ce3cb25cfb
working for in memory single log buffer
2020-04-19 23:37:50 -07:00
Chris Lu
f373232227
Merge pull request #1280 from bukton123/filer_mongodb
...
Add new filer support mongodb
2020-04-19 20:35:19 -07:00
bukton
da37455254
filer mongo add func create index unique directory, name
2020-04-20 08:13:57 +07:00
Chris Lu
4642ed565e
fix tests
2020-04-19 10:42:58 -07:00
bukton
3defa718a1
add default config filer mongodb to scaffold.go
2020-04-19 17:52:38 +07:00
bukton
d6d245dc89
filer mongodb add option poll size and fixed order by list meta
2020-04-19 17:51:32 +07:00
Chris Lu
71ffb98475
broker: add profiling
2020-04-19 03:03:40 -07:00
Chris Lu
d278b4c228
basic pub and sub works!
2020-04-19 00:18:32 -07:00
bukton
082f079ead
filer mongo change func update to insert entry
2020-04-19 11:08:10 +07:00
Chris Lu
788578d4ea
add subscriber api
2020-04-18 16:05:29 -07:00
Chris Lu
767f14dfcd
master, volume default to detected host
2020-04-18 15:20:20 -07:00
Chris Lu
076c8bd3bc
filer master start up with default ip address instead of just localhost
2020-04-18 15:17:27 -07:00
Chris Lu
5d346d44bd
Revert "move go client out"
...
This reverts commit ef1274edc9
.
2020-04-18 12:44:55 -07:00
Chris Lu
ef1274edc9
move go client out
2020-04-18 12:11:50 -07:00
Chris Lu
5af49c7aa5
adjust API
2020-04-18 11:45:57 -07:00
bukton
8c58b0d017
filer mongodb delete folder children
2020-04-19 00:31:49 +07:00
bukton
290c6b7f01
Merge remote-tracking branch 'origin/master' into filer_mongodb
...
# Conflicts:
# go.mod
# go.sum
# weed/server/filer_server.go
2020-04-19 00:21:45 +07:00
bukton
6234ea441b
filer mongodb delete
2020-04-19 00:20:00 +07:00
bukton
24a8192298
filer mongo list
2020-04-19 00:16:50 +07:00
bukton
3c70163798
filer mongodb insert entry and find entry
2020-04-18 23:48:38 +07:00
bukton
cd9cccec55
add import mongodb to filer server
2020-04-18 22:30:13 +07:00
bukton
fd4576651b
add new package mongodb and install lib mongodb
2020-04-18 22:26:57 +07:00
Chris Lu
11f5a6d913
messaging can compile now
2020-04-18 01:12:01 -07:00
Chris Lu
5bea77010f
refactor
2020-04-17 22:39:21 -07:00
Chris Lu
2e2537a9ea
rename files
2020-04-17 22:35:44 -07:00
Chris Lu
e0f5996560
fix "call of Unmarshal passes non-pointer as second argument"
2020-04-17 10:04:41 -07:00
Chris Lu
826f96b6f3
fix unreachable code
2020-04-17 10:04:28 -07:00
Chris Lu
ffa69b24ae
fix travis
2020-04-17 10:01:11 -07:00
Chris Lu
3f3dba5a68
broker: append message logs
2020-04-17 02:29:38 -07:00
Chris Lu
2a45897237
broker: read cipher value from filer
2020-04-17 02:29:00 -07:00
Chris Lu
bda82f61bc
filer: able to append to a file
2020-04-17 02:28:09 -07:00
Chris Lu
722b14337e
add cipher option to meta data updates
2020-04-17 01:26:27 -07:00
Chris Lu
7d3672c60a
simplify
2020-04-17 00:00:48 -07:00
Chris Lu
508f3490a0
update messaging proto
2020-04-16 03:29:57 -07:00
Chris Lu
f5a748d33c
refactoring
2020-04-16 02:55:09 -07:00
Chris Lu
ce4b369be2
scaffolding messaging
2020-04-16 02:21:23 -07:00
Reed
aada5b9aa7
making volume stats size info readable.
2020-04-16 10:08:08 +08:00
Chris Lu
9cacaf5eca
better detect file mime type
2020-04-14 11:32:31 -07:00
Chris Lu
3f1d79512f
only detect when not sure about the types
2020-04-14 11:02:05 -07:00
Chris Lu
d2d1597850
volume: detect gzip effectiveness only when content type is empty
...
revert part of 2286eda575
2020-04-14 10:52:53 -07:00
Chris Lu
23715af402
Merge pull request #1273 from wyxloading/config_race
...
Fix data race about config
2020-04-13 22:24:51 -07:00
Chris Lu
2b5c4fbbf3
tiered caching
...
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
2020-04-13 22:19:27 -07:00
Chris Lu
f282ed444b
refactoring
2020-04-13 21:58:10 -07:00
wuyuxiang
b556d3d035
Fix data race about config
2020-04-14 10:22:40 +08:00
Chris Lu
d8f5985e5e
add logs
2020-04-13 12:58:45 -07:00
Chris Lu
5c9e91393f
1.74
2020-04-13 10:25:48 -07:00
Chris Lu
7764e0465c
refactoring
2020-04-12 21:00:55 -07:00
Chris Lu
8e23dc078b
refactoring
2020-04-12 20:48:21 -07:00
Chris Lu
b062393f3f
refactoring
2020-04-12 17:51:31 -07:00
Chris Lu
5361f999ed
shell: only one shell is allowed to connect to the cluster
...
fix https://github.com/chrislusf/seaweedfs/issues/1266
2020-04-12 17:48:39 -07:00
Chris Lu
eb16bb0287
go mod cleanup
2020-04-12 16:44:58 -07:00
Chris Lu
d30483d642
re-enable system logs
2020-04-12 14:03:07 -07:00
Chris Lu
6f948e4887
remove configurable topics folder location
2020-04-12 13:07:59 -07:00
Chris Lu
94e35cdb35
mount: fix
...
fix bug found by git bisect, but I do not understand why it can cause error!
2020-04-12 03:34:36 -07:00
Chris Lu
1c2e920fa2
fix compilation error
2020-04-12 03:11:19 -07:00
Chris Lu
b5a713be68
filer store: add redis2 redis_cluster2 as default redis store
2020-04-12 02:50:41 -07:00
Chris Lu
e613695af7
a little optimization
2020-04-12 01:13:57 -07:00
Chris Lu
2a1f396df5
avoid duplicated setting chunks into cache
2020-04-12 01:06:50 -07:00
Chris Lu
b9b7da905e
handle nil chunk cache
2020-04-12 01:00:12 -07:00
Chris Lu
211d87cf4c
mount: option to disable caching
2020-04-12 00:52:54 -07:00
Chris Lu
1c65656fb4
s3: add option to fsync buckets
2020-04-11 23:37:10 -07:00
Chris Lu
e4af63a721
volume server: accept fsync=true in write requests
2020-04-11 21:39:16 -07:00
Chris Lu
5987810e5e
set default chunk size to 16
2020-04-11 21:22:52 -07:00
Chris Lu
df97da25f9
mount: add on disk caching
2020-04-11 21:12:41 -07:00
Chris Lu
c8ca234773
refactoring
2020-04-11 14:27:25 -07:00
Chris Lu
df9d538044
rename function
2020-04-11 14:19:44 -07:00
Chris Lu
d7f3acb2c0
refactor
2020-04-11 12:45:24 -07:00
Chris Lu
ed54682598
refactoring
2020-04-11 12:43:17 -07:00
Chris Lu
e909b55633
refactoring
2020-04-11 12:39:46 -07:00
Chris Lu
b7f0ba3800
refactoring
2020-04-11 12:37:41 -07:00
Chris Lu
417125457e
add additional buffers for logs
2020-04-11 02:19:48 -07:00
Chris Lu
4b27afe176
fix sqs message reading
2020-04-10 15:27:47 -07:00
Chris Lu
dce2702e1b
fix the instruction
2020-04-10 14:50:10 -07:00
Chris Lu
13ab1fdaf0
mount: use file default permission 0666
2020-04-10 02:23:23 -07:00
Chris Lu
1101a42e5c
mv : create new folder, move children, and delete old folder
2020-04-10 01:37:03 -07:00
Chris Lu
bcf37346ef
add timestamp inside lock
2020-04-10 01:35:59 -07:00
Chris Lu
9fa065f600
typo in logs
2020-04-09 23:43:09 -07:00
Chris Lu
8db2120bee
add logs
2020-04-09 23:42:59 -07:00
Chris Lu
43c62a8a95
1.73
2020-04-09 21:44:30 -07:00
Chris Lu
eb39df2704
avoid dead loop
2020-04-09 21:43:26 -07:00
Chris Lu
56ec8c4513
fix to avoid nil file an dirty pages
2020-04-09 21:43:05 -07:00
Chris Lu
59f40e2027
volume: best effort to detect ip address
...
fix https://github.com/chrislusf/seaweedfs/issues/1264
2020-04-09 00:26:24 -07:00
Chris Lu
f6a7e79dc3
weed shell: simplify CLI option for filer
2020-04-08 23:57:15 -07:00
Chris Lu
2329d9e0c1
add volume.fix.replication to default scaffold
2020-04-08 23:36:22 -07:00
Chris Lu
bb78ab9915
wait for master to be aware of the failed volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1268
2020-04-08 23:12:37 -07:00
Chris Lu
8764bdb9df
1.72
2020-04-08 22:35:14 -07:00
Chris Lu
8a73276602
refactor a bit
2020-04-08 22:33:36 -07:00
Chris Lu
006f78b70b
make a copy of request data
...
to avoid concurrency issues
2020-04-08 22:31:19 -07:00
Chris Lu
00b993a234
add util for md5
2020-04-08 22:30:38 -07:00
Chris Lu
ed98223b08
still log, but not persisting the changes
2020-04-08 20:32:57 -07:00
Chris Lu
6630541399
ensure correct file size when opening existing files
2020-04-08 12:50:59 -07:00
Chris Lu
b524a40375
add locking to fs cache
2020-04-08 12:50:34 -07:00
Chris Lu
bd56172b82
simplify file handle management
2020-04-08 12:50:20 -07:00
Chris Lu
dc08e4098f
add etag only for PUT or large chunked uploads
2020-04-08 09:13:26 -07:00
Chris Lu
ec2eb8bc48
add If-None-Match and If-Modified-Since
...
fix https://github.com/chrislusf/seaweedfs/issues/1269
2020-04-08 08:12:00 -07:00
Chris Lu
4c498e73de
filer replication: add s3 endpoint
2020-04-07 17:49:00 -07:00
Chris Lu
745f5d2a25
fix setting default value
2020-04-07 01:58:48 -07:00
Chris Lu
54debdc6f7
filer: fix configuration settings
2020-04-07 01:30:53 -07:00
Chris Lu
90d576affe
disable meta data change event logging for now.
2020-04-07 01:05:24 -07:00
Chris Lu
b51d8f03bb
1.71
2020-04-05 22:04:16 -07:00
Chris Lu
5a5908407d
filer: support larger file size
...
fix https://github.com/chrislusf/seaweedfs/issues/1257
2020-04-05 21:48:45 -07:00
Chris Lu
19edd9c091
comment out debug messages
2020-04-05 20:15:17 -07:00
Chris Lu
78ea937606
fix
2020-04-05 17:34:26 -07:00
Chris Lu
53626734d4
iterate through the log buffer
2020-04-05 16:51:30 -07:00
Chris Lu
f71b855db5
adjust caching and notification for recursive directory deletion
2020-04-05 15:03:25 -07:00
Chris Lu
f14b6a09a8
remove duplicated update events
2020-04-05 13:22:15 -07:00
Chris Lu
91da7057b1
refactoring
2020-04-05 13:11:43 -07:00
Chris Lu
2d43f85577
watch entries with common path prefix
2020-04-05 12:51:21 -07:00
Chris Lu
7bc3c93512
add util.PathJoin
2020-04-05 12:40:46 -07:00
Chris Lu
af1f64d244
change from 'weed tail' to 'weed watch'
2020-04-05 12:39:20 -07:00
Chris Lu
b10679fcf0
adjust log file names
2020-04-05 12:38:46 -07:00
Chris Lu
2cb1700ecb
set to 4MB meta log buffer size
2020-04-05 12:38:29 -07:00
Chris Lu
e78ee087e9
fix tests
2020-04-05 01:17:35 -07:00
Chris Lu
dcc6b6a3ed
go fmt
2020-04-05 00:52:31 -07:00
Chris Lu
bf270d9e8c
filer: able to tail meta data changes
2020-04-05 00:51:16 -07:00
Chris Lu
f7b5a1d697
add disk stat to logs
2020-04-04 13:17:23 -07:00
Chris Lu
6e776aaafc
loop through all masters
2020-04-03 00:47:48 -07:00
Chris Lu
4a5e381feb
fail fast if filer configured wrong masters
...
fix https://github.com/chrislusf/seaweedfs/issues/1257
2020-04-03 00:47:33 -07:00
Chris Lu
aaf5e658cc
adjust parameter name
2020-04-03 00:41:05 -07:00
Chris Lu
6c96e51414
re-ordering
2020-04-03 00:40:54 -07:00
Chris Lu
d2f8e50459
make error a little bit more clearer
...
fix https://github.com/chrislusf/seaweedfs/issues/1259
2020-04-03 00:24:09 -07:00
Chris Lu
5f881d2fd5
mount: fail early rather than after mounted the directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1258
2020-04-03 00:21:03 -07:00
Chris Lu
fee3319cbf
handle gzip for image resizing
2020-04-02 23:44:24 -07:00
Chris Lu
73bc286377
volume: passing ext info via URL
2020-04-02 21:23:15 -07:00
Chris Lu
cea52a4faf
volume copying adds cleaning up on error
...
fix https://github.com/chrislusf/seaweedfs/issues/1253
2020-04-02 15:36:55 -07:00
Chris Lu
c981975627
volume.fix.replication: add test cases for complicated moving
...
fix https://github.com/chrislusf/seaweedfs/issues/1253
2020-04-02 02:16:16 -07:00
Chris Lu
ae3d08883f
volume.fsck: fix for erasure encoded volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1254
2020-04-01 22:10:13 -07:00
Chris Lu
c446438ca5
Merge pull request #1255 from levenlabs/ignore
...
Added treat_replication_as_minimums master toml option
2020-04-01 16:48:58 -07:00
James Hartig
eae3f27c80
Added treat_replication_as_minimums master toml option
2020-04-01 19:08:48 -04:00
Chris Lu
2fc2eb74dd
refactoring
2020-03-30 22:57:45 -07:00
Chris Lu
c22094d340
adjust logging
2020-03-30 13:03:43 -07:00
Chris Lu
50a5018b7f
writing meta logs is working
2020-03-30 01:19:33 -07:00
Chris Lu
9dc0b1df8f
refactoring to compile for windows OS
...
windows os does not like to work with fuse
2020-03-29 21:07:55 -07:00
Chris Lu
be6c2f8511
1.70
2020-03-29 13:40:55 -07:00
Chris Lu
54768d0761
fix: delete a file and then create a directory with the same name
2020-03-29 01:39:48 -07:00
Chris Lu
057722bbf4
return part of the chunk if chunkview is not the full chunk
2020-03-29 00:54:39 -07:00
Chris Lu
af6d9976ad
adjust print out
2020-03-28 14:07:25 -07:00
Chris Lu
a75d50bbb8
FUSE: add configurable in memory chunk cache size
2020-03-28 14:07:16 -07:00
Chris Lu
826bc0b7e3
FUSE: add chunk cache for recently accessed file chunks
2020-03-28 13:43:31 -07:00
Chris Lu
4aa82c95e6
refactor
2020-03-28 13:42:35 -07:00
Chris Lu
eedd33dda3
refactoring
2020-03-28 13:41:58 -07:00
Chris Lu
005b4ab3fe
1.69
2020-03-27 04:51:31 -07:00
Chris Lu
d1439c5bd3
fix FUSE read for large files
...
FUSE expects ReadAt do not return partial filled buffer with a nil error.
2020-03-27 04:50:51 -07:00
Chris Lu
f06ca04451
avoid overflow
2020-03-27 04:35:31 -07:00
Chris Lu
e1911760a7
refactoring
2020-03-27 00:30:55 -07:00
Chris Lu
2f5ed29c37
clean up
2020-03-27 00:30:39 -07:00
Chris Lu
2286eda575
sampling whether the data can be gzipped
2020-03-26 23:50:48 -07:00
Chris Lu
5463560912
small test
2020-03-26 23:50:07 -07:00
Chris Lu
34a0d424f8
FUSE mount: report max volume count available disk space
2020-03-26 23:06:21 -07:00
Chris Lu
a5ce359999
refactoring
2020-03-26 22:46:49 -07:00
Chris Lu
0a05052b5c
1.68
2020-03-26 22:22:36 -07:00
Chris Lu
e7e8691720
add some tests
2020-03-26 22:21:52 -07:00
Chris Lu
86398d1efe
S3 API: set deletion to recursive
...
fix https://github.com/chrislusf/seaweedfs/issues/1251
2020-03-26 11:05:54 -07:00
Chris Lu
1fef598347
set target Dir object when moving
2020-03-26 10:56:40 -07:00
Chris Lu
9e2e07b238
tweak
2020-03-26 10:56:18 -07:00
Chris Lu
cbce793753
rename: set new name for renamed files
2020-03-26 03:30:23 -07:00
Chris Lu
c763a492f5
refactor
2020-03-26 03:30:02 -07:00
Chris Lu
712720864d
error checking for remove
2020-03-26 02:21:30 -07:00
Chris Lu
8d9a76005d
add back running output
2020-03-26 02:14:48 -07:00
Chris Lu
dd5b582d05
go fmt
2020-03-26 00:09:01 -07:00
Chris Lu
7f0d87b206
tree structured fs cache
...
FsCache for FsNode, wrapping fs.Node
2020-03-26 00:08:14 -07:00
Chris Lu
2e4fadd10a
fix fscache move
2020-03-25 22:19:19 -07:00
Chris Lu
966df2ae27
purging skip EC volumes
2020-03-25 02:41:22 -07:00
Chris Lu
6d87a0c5f5
batch delete orphan data by file keys
2020-03-25 02:21:15 -07:00
Chris Lu
5ebc95b69b
refactoring
2020-03-25 02:20:19 -07:00
Chris Lu
d79f0a76b1
format output
2020-03-25 00:56:47 -07:00
Chris Lu
d9b56ed88b
shell: volume.fsck add options
2020-03-24 22:38:36 -07:00
Chris Lu
e63a79ade8
better handle lock in case of exception
2020-03-24 18:41:25 -07:00
Chris Lu
4d5554b16f
fix compilation on travis
2020-03-24 18:40:52 -07:00
Chris Lu
d83e648c0b
auto add prefix "fs."
2020-03-24 02:40:51 -07:00
Chris Lu
a875f67a08
fix pwd printout
2020-03-24 02:40:29 -07:00
Chris Lu
00d3f01876
better output format
2020-03-24 02:34:28 -07:00
Chris Lu
0820935290
clean up
2020-03-24 02:20:37 -07:00
Chris Lu
49440cbd2f
shell: add volume.fsck
...
volume.fsck can find out how much space is not used by the filer
2020-03-24 02:18:13 -07:00
Chris Lu
0f10833e0b
fix compilation
2020-03-24 00:16:12 -07:00
Chris Lu
3794f2721f
fix logic
2020-03-24 00:09:12 -07:00
Chris Lu
443b229ef7
disable chunk file output
2020-03-24 00:08:02 -07:00
Chris Lu
bb9b97e2b4
add comments
2020-03-23 23:07:11 -07:00
Chris Lu
8047ec2f51
shell: fs.meta.save add option to export all fileIds for all files
2020-03-23 22:54:46 -07:00
Chris Lu
782d776d2a
refactoring
2020-03-23 22:54:02 -07:00
Chris Lu
38e73463f1
fix du block and byte couting
2020-03-23 21:37:04 -07:00
Chris Lu
b51fa81f0e
fix directory checking
2020-03-23 21:36:39 -07:00
Chris Lu
e666aeece2
simplify parsing filer host and port
2020-03-23 21:26:15 -07:00
Chris Lu
d151185b7e
shell: desupport filer url in the arguments
2020-03-23 20:46:17 -07:00
Chris Lu
40601953bf
comments
2020-03-23 02:54:49 -07:00
Chris Lu
daeb18be30
scaffold: fix master.toml typo
...
fix https://github.com/chrislusf/seaweedfs/issues/1247
2020-03-23 01:45:43 -07:00
Chris Lu
b97768c51c
refactoring
2020-03-23 01:30:22 -07:00
Chris Lu
45ee3736aa
refactoring
2020-03-23 01:25:38 -07:00
Chris Lu
cddcb5cafe
same booleans as before
2020-03-23 01:25:23 -07:00
Chris Lu
f84c288852
purge
2020-03-23 01:14:55 -07:00
Chris Lu
a5aa8be19b
Update webdav_server.go
2020-03-23 01:14:42 -07:00
Chris Lu
7f1e3c843d
refactoring
2020-03-23 01:14:21 -07:00
Chris Lu
c07bcd5065
refactoring
2020-03-23 00:30:02 -07:00
Chris Lu
654a69ff52
refactoring
2020-03-23 00:06:24 -07:00
Chris Lu
c0f0fdb3ba
refactoring
2020-03-23 00:01:34 -07:00
Chris Lu
fbca6b29bd
refactoring
2020-03-22 23:52:55 -07:00
Chris Lu
e39e78ea8d
remove println
2020-03-22 18:37:12 -07:00
Chris Lu
c16dc57a58
fix max volume count reporting
2020-03-22 18:33:19 -07:00
Chris Lu
d6412f27a0
logging
2020-03-22 18:32:56 -07:00
Chris Lu
35208711e5
logging
2020-03-22 18:32:49 -07:00
Chris Lu
e32999108a
add auto configured volume max count help messge
2020-03-22 16:27:15 -07:00
Chris Lu
3137777d83
volume: automatically detect max volume count
2020-03-22 16:21:42 -07:00
Chris Lu
0bf148f49d
logging
2020-03-22 16:19:00 -07:00
Chris Lu
bda13ed593
commented prints
2020-03-22 13:09:51 -07:00
Chris Lu
74e23e8d64
FUSE mount: rename correctly across folders
2020-03-22 13:09:33 -07:00
Chris Lu
64000f5c45
FUSE mount: rename under root
...
fix https://github.com/chrislusf/seaweedfs/issues/1242
2020-03-22 13:09:16 -07:00
Chris Lu
9f2c9b666b
adjust log level
2020-03-22 01:39:08 -07:00
Chris Lu
ae2ee379c0
consistent 64bit size
2020-03-22 01:37:46 -07:00
Chris Lu
2bdd936fb6
purge code
2020-03-22 01:33:40 -07:00
Chris Lu
e93e986727
webdav: stream read
2020-03-22 01:30:27 -07:00
Chris Lu
65d2ea9fb0
FUSE mount: stream read data with buffer
...
fix https://github.com/chrislusf/seaweedfs/issues/1244
2020-03-22 01:00:36 -07:00
Chris Lu
82bfad5b86
refactoring
2020-03-21 23:48:11 -07:00
Chris Lu
6ff9e2835e
filer: skip resizing image if width or height larger than original image
...
fix https://github.com/chrislusf/seaweedfs/issues/1239#issuecomment-602140779
2020-03-21 22:16:00 -07:00
Chris Lu
7c111f7b75
FUSE mount: make "nonempty" optional
...
https://github.com/chrislusf/seaweedfs/issues/1094
2020-03-21 19:14:25 -07:00
Chris Lu
3b3e063f25
1.67
2020-03-20 23:42:49 -07:00
Chris Lu
d3e4a31058
filer: fix where deletion can miss under high concurrency
2020-03-20 23:39:32 -07:00
Chris Lu
cbfe31a9a8
idx file sync before compaction
2020-03-20 23:38:46 -07:00
Chris Lu
3505b06023
report deletion error in the log
2020-03-20 23:38:24 -07:00
Chris Lu
4bc19f1e52
master: fix connections count reporting
2020-03-20 22:41:37 -07:00
Chris Lu
e1b5ac5c04
1.66
2020-03-20 20:39:48 -07:00
Chris Lu
308688c8d0
filer: add back image resizing capability
2020-03-20 20:31:11 -07:00
Chris Lu
f251d03673
refactoring
2020-03-20 15:46:16 -07:00
Chris Lu
11372dd548
refactoring
2020-03-20 15:38:04 -07:00
Chris Lu
c4bea45099
S3 API: fix DeleteMultipleObjectsHandler
...
fix https://github.com/chrislusf/seaweedfs/issues/1241
2020-03-20 14:17:31 -07:00
Chris Lu
165b0d22a4
1.65
2020-03-20 00:05:42 -07:00
Chris Lu
81797a059a
volume: sync volume file right before compaction
...
fix https://github.com/chrislusf/seaweedfs/issues/1237
2020-03-19 23:54:52 -07:00
Chris Lu
709f231e23
tiered storage: add s3 endpoint for private s3 implementation
...
fix https://github.com/chrislusf/seaweedfs/issues/1238
2020-03-19 21:13:56 -07:00
Chris Lu
d848d08944
use public url also for deletions
2020-03-18 11:16:45 -07:00
Chris Lu
6566c8e114
weed upload: add usePublicUrl option
...
fix https://github.com/chrislusf/seaweedfs/issues/1236
2020-03-18 10:50:53 -07:00
Chris Lu
db9854fa0a
1.64
2020-03-17 10:58:35 -07:00
Chris Lu
bec6ec7db6
go fmt
2020-03-17 10:01:55 -07:00
Chris Lu
b964bbab3d
fix compilation
2020-03-17 10:01:24 -07:00
Chris Lu
c3cb6fa1d7
volume: compaction can cause readonly volumes
...
address https://github.com/chrislusf/seaweedfs/issues/1233
2020-03-17 09:43:57 -07:00
Chris Lu
f9b3750ad1
volume: handle repeated writes response
2020-03-15 04:16:00 -07:00
Chris Lu
22400c6633
consistent gzip logic
...
local store gzip same as replicated writes
2020-03-15 04:15:40 -07:00
Chris Lu
560df51def
refactoring
2020-03-15 03:11:26 -07:00
Chris Lu
7edbee6f57
volume: proxy writes to remote volume server, with replication or not
...
the panic is triggered by uploading a file to a volume server not holding the designated replica.
2020-03-15 10:20:14.365488 I | http: panic serving 127.0.0.1:57124: runtime error: invalid memory address or nil pointer dereference
goroutine 119 [running]:
net/http.(*conn).serve.func1(0xc0001a8000)
/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1772 +0x139
panic(0x2316fe0, 0x3662900)
/home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/panic.go:973 +0x396
github.com/chrislusf/seaweedfs/weed/topology.getWritableRemoteReplications(0xc00009c000, 0x2, 0x7ffeefbffbd2, 0xe, 0x0, 0xa, 0x0, 0x0, 0xbb4bf1f7)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:157 +0x53
github.com/chrislusf/seaweedfs/weed/topology.ReplicatedWrite(0x7ffeefbffbd2, 0xe, 0xc00009c000, 0xc000000002, 0xc000472750, 0xc0001b2200, 0x0, 0x1, 0x0)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:29 +0xc7
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).PostHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers_write.go:52 +0x56f
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).privateStoreHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers.go:37 +0x21f
net/http.HandlerFunc.ServeHTTP(0xc0004420e0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2012 +0x44
net/http.(*ServeMux).ServeHTTP(0xc0001fc800, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2387 +0x1a5
net/http.serverHandler.ServeHTTP(0xc0001781c0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0001a8000, 0x2934420, 0xc000212400)
/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2933 +0x35c
Eg:
server A (datacenter 1) and server B (datacenter 2) hold replica (100) for volume 1.
If you upload a file with a key 1,xxxxx to server C (datacenter 3) will trigger the panic on server C.
The server C should either proxy upload file to the correct volume server or should return an HTTP error code and not panic.
2020-03-15 02:50:42 -07:00
Chris Lu
7b37178716
filer: close stores if interrupted
2020-03-14 20:30:26 -07:00
Chris Lu
d022b6bc0e
fix compilation
2020-03-14 16:32:16 -07:00
Chris Lu
0576a27f44
protect against possible nil
...
which is unlikely to happen though
2020-03-14 16:06:06 -07:00
Chris Lu
cc52e8ca87
1.63
2020-03-14 14:25:07 -07:00
Chris Lu
ee635dcc36
refactoring
2020-03-14 14:08:00 -07:00
Chris Lu
18cda6adbb
cleaner fix
2020-03-14 13:55:32 -07:00
Chris Lu
17efa361f9
fix possible nil
2020-03-14 13:53:03 -07:00
Chris Lu
91e0a987e4
1.62
2020-03-14 01:03:59 -07:00
Chris Lu
e2e691d9c2
clean up, add test
2020-03-13 23:53:15 -07:00
Chris Lu
3cc9e85895
volume: vacuum pass preallocate variable
2020-03-13 16:17:44 -07:00
Chris Lu
4ce23e8809
fix compilation error
2020-03-13 15:51:38 -07:00
Chris Lu
c90eb0da1f
volume: handling readonly volumes after compaction
...
ensure readonly volumes are not added as writable
2020-03-13 15:41:27 -07:00
Chris Lu
e85da50ad4
remove tikv, move its support to "tikv" branch
2020-03-11 20:59:00 -07:00
Chris Lu
d439d83772
volume: follow compactionBytePerSecond
...
related to https://github.com/chrislusf/seaweedfs/issues/1108
2020-03-11 10:32:17 -07:00
Chris Lu
4237a813cc
skip printouts
2020-03-11 01:13:54 -07:00
Chris Lu
ebc739afb5
filer store postgres: skip empty parameters
...
better fix https://github.com/chrislusf/seaweedfs/issues/1227
2020-03-11 01:13:54 -07:00
HongyanShen
81610ed006
fix : #1226
2020-03-11 14:37:14 +08:00
Chris Lu
60f5f05c78
1.61
2020-03-10 13:52:49 -07:00
Chris Lu
7f3ce7b6d6
adjust tikv build env
2020-03-10 13:36:45 -07:00
Chris Lu
d3e1ef64b3
ensure error checking on windows
2020-03-10 13:33:26 -07:00
Chris Lu
88a110e67e
release file handle
2020-03-10 13:32:22 -07:00
Chris Lu
43e6261579
volume: fix memory leak
2020-03-10 13:31:23 -07:00
Chris Lu
d214cefc2e
filer: list diretory pagination make up for the expired entries
2020-03-09 23:28:01 -07:00
Chris Lu
e6de42f888
1.60
2020-03-09 22:32:40 -07:00
Chris Lu
e73b0c2457
remove not ready status
2020-03-09 22:31:14 -07:00
Chris Lu
0871d2cff0
volume: fix memory leak during compaction
...
fix https://github.com/chrislusf/seaweedfs/issues/1222
2020-03-09 22:29:02 -07:00
Chris Lu
14ae33d642
adjust logging
2020-03-09 10:36:03 -07:00
Chris Lu
89eb05b50f
filer: support TTL for all filer stores
2020-03-09 01:02:01 -07:00
Chris Lu
8a899992f2
filer: fix ttl parsing
...
fix https://github.com/chrislusf/seaweedfs/issues/1225
2020-03-09 00:16:10 -07:00
Chris Lu
c32f95c380
1.59
2020-03-08 22:02:34 -07:00
Chris Lu
3934c69757
clean up
2020-03-08 21:54:47 -07:00
Chris Lu
2e3f6ad3a9
filer: remember content is gzipped or not
2020-03-08 21:39:33 -07:00
Chris Lu
5ac6297c68
adjust parameter names
2020-03-08 17:03:37 -07:00
Chris Lu
e04c1a1361
filer: remove chunk manifest file support
...
This is not needed for filer
2020-03-08 17:03:27 -07:00
Chris Lu
1dc30214cb
mark encryptVolumeData as not ready
2020-03-08 17:02:47 -07:00
Chris Lu
9b3109a5d8
filer: processing all response headers, no pass through to volume server
...
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
2020-03-08 15:42:44 -07:00
Chris Lu
11fceaf2f7
refactoring
2020-03-07 18:09:30 -08:00
Chris Lu
1a75fbfce0
refactoring
2020-03-07 18:06:48 -08:00
Chris Lu
afb20de14c
breaks dependency loop
2020-03-07 17:01:39 -08:00
Chris Lu
8645283a7b
fuse mount: avoid lookup nil entry
...
fix https://github.com/chrislusf/seaweedfs/issues/1221
2020-03-07 16:51:46 -08:00
Chris Lu
936e7cdbfb
pass in filer.copy cipher option
2020-03-07 12:46:00 -08:00
Chris Lu
9228ff192c
encryption works
2020-03-07 11:08:57 -08:00
Chris Lu
0375ce2c2e
filer: set mime type from volume server
2020-03-07 07:25:15 -08:00
Chris Lu
1ae83c2938
go fmt
2020-03-07 06:12:57 -08:00
Chris Lu
564629444b
format
2020-03-07 06:12:20 -08:00
Chris Lu
8c3e25b38a
mimee type default "application/octet-stream" to empty string
2020-03-07 06:08:08 -08:00
Chris Lu
dba35404e4
filer: HEAD response add content type
2020-03-07 06:08:08 -08:00
Chris Lu
ea1169dc80
filer cipher: single chunk http POST and PUT and read
2020-03-07 06:08:08 -08:00
Chris Lu
e3b8bf5588
Merge pull request #1219 from song-zhang/master
...
schedule new volume by free volume number of nodes
2020-03-07 05:54:06 -08:00
zhangsong
40f70481cd
schedule new volume by free volume number of nodes
2020-03-07 21:41:42 +08:00
Chris Lu
13e215ee5c
filer: option to encrypt data on volume server
2020-03-06 00:49:47 -08:00
Chris Lu
31c481e3fc
fix typo
2020-03-05 16:00:20 -08:00
Chris Lu
465d18930b
filer redis: directory listing follow ttl
...
fix https://github.com/chrislusf/seaweedfs/issues/1217
2020-03-05 10:35:21 -08:00
Chris Lu
e031605248
purge code
2020-03-04 00:44:27 -08:00
Chris Lu
f90c43635d
refactoring
2020-03-04 00:39:47 -08:00
Chris Lu
9b7dae8037
filer: fix uploaded chunked file size reporting bug.
2020-03-02 20:27:14 -08:00
Chris Lu
7335e62199
volume: PUT also conditionally gzip compress
2020-03-01 23:37:02 -08:00
Chris Lu
410bce3925
go fmt
2020-03-01 22:39:08 -08:00
Chris Lu
dc40413847
fix error
2020-03-01 22:21:13 -08:00
Chris Lu
6a8484b4ae
master able to list all master clients by type
2020-03-01 22:13:47 -08:00
Chris Lu
0ca68a2a6d
WIP
2020-03-01 22:13:15 -08:00
Chris Lu
4532640ffd
1.58
2020-03-01 17:59:12 -08:00
Chris Lu
151114ff59
volume: fix readonly status reporting
2020-03-01 17:57:37 -08:00
Chris Lu
1df13d0d6d
adjust instructions
2020-02-27 13:15:21 -08:00
Chris Lu
f87f2045b3
delete old file first on windows
...
fix https://github.com/chrislusf/seaweedfs/issues/1210
2020-02-27 00:59:35 -08:00
Chris Lu
757c7d67ad
avoid nil DataBackend
2020-02-27 00:07:25 -08:00
Chris Lu
555413d9fc
weed queue starts
2020-02-27 00:07:13 -08:00
Chris Lu
430b5a49e2
unused code
2020-02-26 19:35:00 -08:00
Chris Lu
8c1da71402
remove dead code
2020-02-26 17:27:49 -08:00
Chris Lu
ed0acd1722
go fmt
2020-02-26 16:52:57 -08:00
Chris Lu
37a3628b2e
grpc add PermitWithoutStream to client and server options
2020-02-26 16:51:38 -08:00
Chris Lu
556dd76303
s3: wait to connect to filer
2020-02-26 16:49:47 -08:00
Chris Lu
0156e2975a
mount: add mode to run external to SeaweedFS container cluster
2020-02-26 16:46:01 -08:00
Chris Lu
543cf1c80e
fix compilation error
2020-02-25 23:59:30 -08:00
Chris Lu
b1f377f822
mount: fail fast when connecting to filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1034
2020-02-25 23:51:34 -08:00
Chris Lu
36d6595658
remove ctx
2020-02-25 22:38:27 -08:00
Chris Lu
fd9612d66e
remove ctx
2020-02-25 22:37:54 -08:00
Chris Lu
86cce3eb58
fix test
2020-02-25 22:29:01 -08:00
Chris Lu
97ab8a1976
remove ctx if possible
2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9
avoid reusing context object
...
fix https://github.com/chrislusf/seaweedfs/issues/1182
2020-02-25 21:50:12 -08:00
Chris Lu
bd3254b53f
adjust logging
2020-02-25 17:24:08 -08:00
Chris Lu
0841bedb15
move filer assign volume grpc errror to response
2020-02-25 17:15:09 -08:00
Chris Lu
bc38b72a20
s3: implemented DeleteMultipleObjects
2020-02-25 14:38:36 -08:00
Chris Lu
35dde56711
refactoring
2020-02-25 12:58:45 -08:00
Chris Lu
986d63cd5e
s3: increase list objects or parts limit from 1000 to 10000
...
10x of Amazon S3 limits
2020-02-25 11:45:40 -08:00
Chris Lu
7d10fdf737
fix directory lookup nil
2020-02-25 11:13:06 -08:00
Chris Lu
e86da5a491
minor
2020-02-25 00:42:48 -08:00
Chris Lu
0644d63748
shell: add commands for bucket
2020-02-24 23:30:01 -08:00
Chris Lu
6ab7368ef2
filer: dynamically create bucket under /buckets folder
2020-02-24 22:28:45 -08:00
Chris Lu
5bcb44eda9
filer: move buckets folder to filer.toml since it should not be changed often
2020-02-24 14:42:57 -08:00
Chris Lu
d8dec2323b
s3: move buckets folder configuration to filer
2020-02-24 14:34:14 -08:00
Chris Lu
b06b7ca6e6
adjust UI
2020-02-23 22:01:03 -08:00
Yoni Nakache
0d60bb4427
display bug, header is Free but data was Used
2020-02-24 07:47:27 +02:00
Chris Lu
e96f85ed8a
make pb files
2020-02-23 18:06:34 -08:00
Chris Lu
346a1cf0b9
Merge pull request #1207 from LazyDBA247-Anyvision/update-stats-and-status-volume-server
...
Update stats and status volume server
2020-02-23 18:05:25 -08:00
Chris Lu
bbb163c5ad
add for https://github.com/chrislusf/seaweedfs/pull/1207
2020-02-23 18:04:31 -08:00
LazyDBA247-Anyvision
4ff513d64d
staus route: add DiskStatuses for disk in the volume server status
...
whem monitoring server, better to know the status of the disks & volumes in a single route.
2020-02-23 23:27:09 +02:00
LazyDBA247-Anyvision
ab91dbe1d7
fillInDiskStatus: update function to calculate new stats
2020-02-23 23:25:56 +02:00
LazyDBA247-Anyvision
ec8619176a
volume server protobuf: add precentFree & percentUsage to the DiskStatus
2020-02-23 23:24:24 +02:00
Yoni Nakache
cd4c9a365b
DiskStats: adding Total & Percent Usage
...
making relevant data visible and readable.
2020-02-23 22:33:47 +02:00
Chris Lu
6ea394b802
1.57
2020-02-22 21:34:49 -08:00
Chris Lu
4ed6b584e2
s3: access control limited by bucket
2020-02-22 21:34:18 -08:00
Chris Lu
e83bfd0a35
adjust log level
2020-02-22 21:23:30 -08:00
Chris Lu
48b7ad5fa8
s3: deny anonymous type
2020-02-22 14:01:04 -08:00
Chris Lu
f55f49970f
move volume server stats to grpc
2020-02-21 21:45:03 -08:00
Chris Lu
3a1d017de2
shell: ensure dc and rack does not change for replicated volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1203
2020-02-21 21:23:25 -08:00
Chris Lu
621cdbdf58
filer: avoid possible timeouts for updates and deletions
2020-02-20 15:44:17 -08:00
Chris Lu
45156cc2fe
add test cases for replication=011
2020-02-20 01:21:11 -08:00
Chris Lu
b07089cff2
webdav out of "unstable"
2020-02-18 19:38:59 -08:00
Chris Lu
dbabdd418e
add unhandled request auth type
...
fix
2020-02-18 11:43:57.396699 I | http: panic serving 172.28.0.43:50658: runtime error: invalid memory address or nil pointer dereference
goroutine 595 [running]:
net/http.(*conn).serve.func1(0xc0001fe3c0)
/usr/lib/go/src/net/http/server.go:1767 +0x13b
panic(0x55c4e35f3820, 0x55c4e48b3c40)
/usr/lib/go/src/runtime/panic.go:679 +0x1b6
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).authRequest(0xc0004b84e0, 0xc000115900, 0xc0000bb650, 0x1, 0x1, 0x55c4e399d740)
/go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:143 +0x11c
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).Auth.func1(0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
/go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:111 +0x5e
net/http.HandlerFunc.ServeHTTP(0xc0004b87e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
/usr/lib/go/src/net/http/server.go:2007 +0x46
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0004ba000, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
/root/go/pkg/mod/github.com/gorilla/mux@v1.7.3/mux.go:212 +0xe4
net/http.serverHandler.ServeHTTP(0xc00011e0e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
/usr/lib/go/src/net/http/server.go:2802 +0xa6
net/http.(*conn).serve(0xc0001fe3c0, 0x55c4e399d680, 0xc000894180)
/usr/lib/go/src/net/http/server.go:1890 +0x877
created by net/http.(*Server).Serve
/usr/lib/go/src/net/http/server.go:2927 +0x390
2020-02-18 09:16:04 -08:00
Chris Lu
443a68484a
1.56
2020-02-17 12:54:45 -08:00
Chris Lu
0a57db7026
Merge pull request #1200 from chrislusf/add_s3_acl
...
Add s3 ACL
2020-02-17 12:53:30 -08:00
Chris Lu
4cdde5f569
configuration stores the identity list
2020-02-17 12:31:59 -08:00
Chris Lu
9232d3ac68
ignore error when counting closed connections
2020-02-14 10:28:02 -08:00
Chris Lu
96c1ae8471
refactoring the close http response
2020-02-14 09:46:36 -08:00
Chris Lu
cf5064d702
properly close http response
2020-02-14 09:09:15 -08:00
Chris Lu
9b6296e77a
volume: add grpc file read operation
...
This is added more for performance benchmarking.
2020-02-14 00:37:32 -08:00
panyc16
7b3764fd9e
make loadExistingVolume() return bool
2020-02-13 15:19:00 +08:00
Chris Lu
62c34454d8
1.55
2020-02-11 23:00:31 -08:00
Chris Lu
83836f4299
filer: recursively batch delete file chunks
...
fix https://github.com/chrislusf/seaweedfs/issues/1197
2020-02-11 22:54:10 -08:00
Chris Lu
2a6db0fd43
volume: return 204 for unchanged file uploads
...
fix https://github.com/chrislusf/seaweedfs/issues/1196
2020-02-11 09:45:02 -08:00
Chris Lu
33b3bd467c
Revert "HEAD operation changes to fasthttp"
...
This reverts commit 58f126fd27
.
2020-02-10 13:43:53 -08:00
Chris Lu
58f126fd27
HEAD operation changes to fasthttp
2020-02-10 09:13:29 -08:00
Chris Lu
29945fad51
1.54
2020-02-09 18:07:17 -08:00
Chris Lu
b4abe3c081
unused
2020-02-09 18:02:17 -08:00
Chris Lu
f3ce3166ad
add streaming v4
2020-02-09 17:42:17 -08:00
Chris Lu
b90ad6f452
add v2 support
2020-02-09 16:02:05 -08:00
Chris Lu
e7b63d50b1
add s3.config to server.go
2020-02-09 14:31:51 -08:00
Chris Lu
9ed364f053
support acl
2020-02-09 14:30:02 -08:00
Chris Lu
8d94564f41
refactor
2020-02-04 21:16:34 -08:00
Chris Lu
c1288e9eb4
volume: sdx generation uses memdb instead of compactMap
...
fix https://github.com/chrislusf/seaweedfs/issues/1194
2020-02-04 21:12:09 -08:00
Chris Lu
08e4702542
sync before closing index file
...
merge from 0181f87d91
2020-02-04 10:38:12 -08:00
Chris Lu
3a35632d58
sync before closing file
...
merge changes from 1747fc2d52
2020-02-04 10:37:14 -08:00
Chris Lu
dc786a63ac
master: add configurable volume growth toml setting
2020-02-03 18:15:16 -08:00
Chris Lu
382ff218d3
filer: set file size, streaming chunk file uploading
...
fix https://github.com/chrislusf/seaweedfs/issues/1193
2020-02-03 17:04:06 -08:00
Chris Lu
40ae533fa3
shell: add volume.configure.replication to change replication for a volume
...
fix https://github.com/chrislusf/seaweedfs/issues/1192
2020-02-02 15:37:23 -08:00
Chris Lu
fb19263a71
fix build
2020-01-31 00:59:48 -08:00
Chris Lu
beb0a1599e
1.53
2020-01-31 00:32:09 -08:00
Chris Lu
b2743afaee
remove println
2020-01-31 00:21:18 -08:00
Chris Lu
a80ecbfe84
s3: add s3 copy
...
fix https://github.com/chrislusf/seaweedfs/issues/1190
2020-01-31 00:11:12 -08:00
Chris Lu
3b043ead49
Merge pull request #1189 from eryx67/compression-wav
...
Support wav files compression
2020-01-29 22:37:35 -08:00
Chris Lu
abf90ad7b7
remove testing for viper
2020-01-29 21:38:53 -08:00
Chris Lu
20b1fb9ab7
fix test
2020-01-29 21:24:06 -08:00
eryx67
2a87087590
support wav files compression
2020-01-30 10:11:47 +05:00
Chris Lu
8925f3305d
adjust example
2020-01-29 09:11:07 -08:00
Chris Lu
d335f04de6
support env variables to overwrite toml file
2020-01-29 09:09:55 -08:00
Chris Lu
27b94cb65b
fix wrong url
...
fix https://github.com/chrislusf/seaweedfs/issues/1187
2020-01-28 00:49:47 -08:00
Chris Lu
f724057051
reduce logs
2020-01-27 21:20:41 -08:00
Chris Lu
c8d543ef51
avoid retry delay
2020-01-27 00:58:26 -08:00
Chris Lu
081bc1ea25
mount: able to write to any part of a file
2020-01-27 00:54:52 -08:00
Chris Lu
2f6bb57979
logging
2020-01-27 00:54:21 -08:00
Chris Lu
234f69452b
1.52
2020-01-26 21:04:40 -08:00
Chris Lu
72a64a5cf8
use the same context object in order to retry
2020-01-26 14:42:11 -08:00
Chris Lu
08e4b56a8a
mount: able to handle large git clone
2020-01-26 13:01:11 -08:00
Chris Lu
19a05ad174
add test cases
2020-01-26 12:47:23 -08:00
Chris Lu
835da19c09
add logging
2020-01-26 00:50:18 -08:00
Chris Lu
c48fc8b4de
grpc send error via response instead of grpc error
2020-01-25 09:17:19 -08:00
Chris Lu
ccf3859152
log
2020-01-25 08:16:43 -08:00
Chris Lu
4433c7bdca
logs
2020-01-25 07:34:09 -08:00
Chris Lu
8cccccce9f
mount: reuse the entry object for the new directory
...
avoid mkdir and then query for the same directory
reduces these "context canceled" issues
attr read directory:"/seaweedfs/other/java/s3copier/src/main/java/com" name:"seaweedfs" : rpc error: code = Canceled desc = context canceled
2020-01-25 01:15:54 -08:00
Chris Lu
90e30c0170
log
2020-01-25 00:32:18 -08:00
Chris Lu
8a42aa8221
adjust logging
2020-01-25 00:31:53 -08:00
Chris Lu
28d1b34856
logging
2020-01-24 22:13:06 -08:00
Chris Lu
6e25acc681
add logging
2020-01-24 22:01:51 -08:00
Chris Lu
b6e6ca8595
adjust logs
2020-01-24 20:06:58 -08:00
Chris Lu
d4cde5df49
return empty response if not found
2020-01-24 18:07:34 -08:00
Chris Lu
3cdb27fafc
adjust logging
2020-01-24 17:55:39 -08:00
Chris Lu
6107509c64
adjust logging
2020-01-24 17:26:18 -08:00
Chris Lu
e6f9f5da3a
adjust logging
2020-01-24 17:26:03 -08:00
Chris Lu
24f62ebe76
minor changes
2020-01-24 10:02:53 -08:00
Chris Lu
5f1109f143
reduce log
2020-01-24 01:41:51 -08:00
Chris Lu
9d93c83654
mount: return IO error for mkdir
2020-01-24 01:41:31 -08:00
Chris Lu
107e8a56ea
retry context canceled request
2020-01-24 01:40:51 -08:00
Chris Lu
2f75264ec7
mount: avoid use uint64 for greater than zero test
2020-01-23 22:12:57 -08:00
Chris Lu
c936a12afa
mount: update the file size so file Attr() can read
...
if file Attr() during file writes, the OS, at least for Mac, may try to resend the data to write, which usually are empty.
2020-01-23 21:59:58 -08:00
Chris Lu
c2e589f202
mount: better combines connected intervals to write to volume servers
2020-01-22 23:00:04 -08:00
Chris Lu
6a5c037099
fix http range requests
2020-01-22 22:59:12 -08:00
Chris Lu
72e1aff237
add lock on dirty pages
2020-01-22 15:38:25 -08:00
Chris Lu
66ded8804a
adjust logging
2020-01-22 15:37:59 -08:00
Chris Lu
6b48d246a5
mount: read data that is just written
...
able read on data not flushed
multiple file open shares the same file handle
fix https://github.com/chrislusf/seaweedfs/issues/1182 on linux
2020-01-22 13:42:03 -08:00
Chris Lu
09f4ceef3a
mount: reuse previous file node
2020-01-22 11:43:43 -08:00
Chris Lu
d4e75a0d18
filer: option to create file only if it is new, O_EXCL
2020-01-22 11:42:40 -08:00
Chris Lu
9b01a99d9a
adjust logging
2020-01-21 22:45:50 -08:00
Chris Lu
c8b2dac6c1
volume: avoid sharing volume dat file handle
...
possibly help on https://github.com/chrislusf/seaweedfs/issues/1184
2020-01-21 21:18:01 -08:00
Chris Lu
bb1be61602
1.51
2020-01-20 20:30:03 -08:00
Chris Lu
fca6152c3a
fix print format
2020-01-20 20:24:23 -08:00
Chris Lu
a990ef2106
mount: fix problems found in issue 1182
...
fix https://github.com/chrislusf/seaweedfs/issues/1182
always use the non-duplicated fs.Node
Forget() the fs.Node
Rename will also use the right fs.Node
Avoid using the same file handle for the same file
2020-01-20 20:21:01 -08:00
Chris Lu
630f72f8c5
mount: able to trim a file
2020-01-20 00:00:08 -08:00
Chris Lu
1b0bfbaf59
refactoring
2020-01-19 23:59:46 -08:00
Chris Lu
2f15e93466
mount: rename also clear the cache
...
fix https://github.com/chrislusf/seaweedfs/issues/1182
2020-01-19 12:31:56 -08:00
Chris Lu
04019aa161
mount: deletion ignore old vid
2020-01-19 12:07:26 -08:00
Chris Lu
6a40c688e0
mount: modify file size
2020-01-19 12:07:04 -08:00
Chris Lu
39edcfde4c
filer: set inode
...
avoid "getcwd: No such file or directory"
2020-01-19 12:06:19 -08:00
Chris Lu
e9bb0c179f
adjust logs
2020-01-18 13:25:17 -08:00
Chris Lu
061a61e018
logs
2020-01-15 19:09:00 -08:00
Chris Lu
230a0e4a33
filer: no entry if not found
2020-01-15 19:08:54 -08:00
Chris Lu
e4b660699f
correct comments
...
fix https://github.com/chrislusf/seaweedfs/issues/1181
2020-01-12 20:31:33 -08:00
Chris Lu
d1ab16b6e3
treat it as a single node cluster if empty raft server name
...
possible fix for https://github.com/chrislusf/seaweedfs/issues/1118
2020-01-10 00:37:44 -08:00
Chris Lu
30c7148020
1.50
2020-01-08 22:08:35 -08:00
Chris Lu
8dd260623e
protect against nil deletion results
...
fix https://github.com/chrislusf/seaweedfs/issues/1180
2020-01-08 21:50:59 -08:00
Chris Lu
89e16bd2e8
skip error when draining reader
...
fix https://github.com/chrislusf/seaweedfs/issues/1179
2020-01-08 18:07:07 -08:00
Chris Lu
66ab09aa4a
1.49
2020-01-08 09:51:35 -08:00
Chris Lu
f8a20ef35e
add the old way to compact as a comment
2020-01-08 09:45:42 -08:00
Chris Lu
288baf37fd
saving .vif files correctly
2020-01-08 09:45:26 -08:00
Chris Lu
acf7ca7b93
volume: fix compaction
2020-01-08 09:45:03 -08:00
Chris Lu
943f4986ef
fix possible nil
2020-01-08 08:49:18 -08:00
Chris Lu
9995d3bcb5
remove println
2020-01-07 21:49:28 -08:00
Chris Lu
dc9457fb87
fix compilation error
2020-01-07 09:03:47 -08:00
Chris Lu
aa73364a24
weed fix: add back .idx file generation
2020-01-06 16:29:59 -08:00
Chris Lu
4e731f1c8b
volume: copy volumes also include .vif file
2020-01-04 11:28:29 -08:00
Chris Lu
df636e4c14
remove .vif files
2020-01-03 12:46:42 -08:00
Chris Lu
3eafec4b29
volume: add option to limit file size
2020-01-03 00:37:26 -08:00
Chris Lu
9c139e0601
update help message
2020-01-01 12:38:29 -08:00
Chris Lu
88a80ab557
fix help message
2020-01-01 12:37:38 -08:00
Chris Lu
aa1807e082
1.48
2020-01-01 09:27:02 -08:00
Chris Lu
86d4b18eb5
filer: optional recursive deletion
...
fix https://github.com/chrislusf/seaweedfs/issues/1176
2019-12-31 11:52:54 -08:00
Chris Lu
09043c8e5a
fs: synchronized meta file writing
...
fix https://github.com/chrislusf/seaweedfs/issues/1175
2019-12-29 20:19:51 -08:00
Chris Lu
509f314350
keep .vif file for ec to normal volume conversion
2019-12-28 21:52:55 -08:00
Chris Lu
672868b460
always create .vif file
2019-12-28 21:52:06 -08:00
Chris Lu
03ab0b1b9a
correct .vif file creation
2019-12-28 21:40:12 -08:00
Chris Lu
b6f7033478
go fmt
2019-12-28 21:37:29 -08:00
Chris Lu
9134092a8d
endure creating .vif files
2019-12-28 21:36:15 -08:00
Chris Lu
0cc8347a3a
go fmt
2019-12-28 21:13:10 -08:00
Chris Lu
8fbc0a9163
fix edge cases
2019-12-28 12:59:31 -08:00
Chris Lu
37b64a50b4
ec: generate and copy .vif file
2019-12-28 12:44:59 -08:00
Chris Lu
c06f7eb48a
load volume info from .vif file, use superblock as a backup
2019-12-28 12:28:58 -08:00
Chris Lu
f4a74e03d1
refactoring: separating .vif from tier file loading
2019-12-28 11:35:27 -08:00
Chris Lu
2000284435
rename volume tier info to volume info
2019-12-28 11:21:49 -08:00
Chris Lu
e8b357fd53
rename from .tier to .vif
2019-12-28 11:17:39 -08:00
Chris Lu
b3c83f9404
fix spelling
2019-12-28 11:16:10 -08:00
Chris Lu
1fd8926ac7
ignore draining error
2019-12-25 21:39:33 -08:00
Chris Lu
48d28d3eb2
tier: support remote file attributes and remember the file extension
2019-12-25 21:37:24 -08:00
divinerapier
4cbb6fa199
feat: drains http body if buffer is too small
...
Signed-off-by: divinerapier <poriter.coco@gmail.com>
2019-12-26 11:28:43 +08:00
divinerapier
84640d07b7
fix: handle errors for ReadUrl
...
Signed-off-by: divinerapier <poriter.coco@gmail.com>
2019-12-26 10:07:07 +08:00
Chris Lu
1346437d71
Merge pull request #1171 from stlpmo-jn/redundant_type_conversion
...
remove the redundant type conversion
2019-12-25 17:02:05 -08:00
Chris Lu
35393b4a02
cleanly destroy remote files
2019-12-25 16:17:58 -08:00
Chris Lu
eecad6062a
go fmt
2019-12-25 09:56:42 -08:00
Chris Lu
d960b3474a
tier storage: support downloading the remote dat files
2019-12-25 09:53:13 -08:00
stlpmo
f3de4b6c18
remove the redundant type conversion
2019-12-25 10:13:45 +08:00
Chris Lu
3ebeae0c0b
ec encode distribute ec data and parity shards evenly
2019-12-24 16:52:21 -08:00
Chris Lu
9ff72f616a
go fmt
2019-12-24 14:56:16 -08:00
Chris Lu
efd2f50ede
compaction changed to .idx based deletion
2019-12-24 14:55:50 -08:00
Chris Lu
7ba6be2312
volume ui add readonly status
2019-12-24 14:55:26 -08:00
Chris Lu
d8b39fe92a
testing
2019-12-24 11:29:26 -08:00
Chris Lu
f42b5bd0f5
rename .sdb to .sdx
2019-12-24 10:19:12 -08:00
Chris Lu
abffe857a1
change btree map to in memory level db
2019-12-24 10:18:56 -08:00
Chris Lu
72a561ab7c
refactoring
2019-12-24 08:13:01 -08:00
Chris Lu
9114f7f432
Merge pull request #1169 from stlpmo-jn/http_304_has_no_resp_body
...
remove the http 304 response body
2019-12-24 07:08:46 -08:00
Chris Lu
409a3fe41f
Merge pull request #1168 from stlpmo-jn/optimize_DeleteCollectionFromDiskLocation
...
decouple the volume.Destroy() from the operation of unmountVolume()
2019-12-24 07:08:04 -08:00
Chris Lu
37a836c8a7
Merge pull request #1167 from stlpmo-jn/volume_server_core_dump
...
resolve issue : create volume on no free space disk, the volume server core dump
2019-12-24 07:07:41 -08:00
stlpmo
f87ccbddb4
remove the http 304 response body
2019-12-24 17:56:50 +08:00
stlpmo
38e4b79125
decouple the volume.Destroy() from the operation of unmountVolume()
2019-12-24 17:20:34 +08:00
stlpmo
f300c35266
resolve issue : when the host disk full, the volume server will core dump
2019-12-24 16:36:15 +08:00
Chris Lu
1ad34a2487
ed.decode prefers servers with most data shards
2019-12-24 00:00:45 -08:00
Chris Lu
a18f62bbe7
only copy required shards
2019-12-23 18:06:13 -08:00
Chris Lu
c7d5a0b00c
adjust help message
2019-12-23 17:58:47 -08:00
Chris Lu
8a2d529759
go fmt
2019-12-23 12:48:53 -08:00
Chris Lu
09ca936c78
shell: add ec.decode command
2019-12-23 12:48:20 -08:00
Chris Lu
dda5c6d3cb
fmt
2019-12-22 04:31:36 -08:00
Chris Lu
a9b1ee18af
default redis reads options
...
details in https://github.com/chrislusf/seaweedfs/pull/1165
2019-12-21 09:30:51 -08:00
Chris Lu
fcbd04c425
default to false in scaffold.go
2019-12-21 08:31:27 -08:00
Chris Lu
62b1a346ef
set default
2019-12-20 20:57:38 -08:00
Chris Lu
cd47112c40
Merge pull request #1165 from dqsully/master
...
Add readOnly and routeByLatency options to Redis Cluster Store
2019-12-20 20:56:34 -08:00
Chris Lu
573b74f24f
add SetDefault function for Configuration
2019-12-20 20:56:14 -08:00
Dakota Sullivan
f934a85d58
Add readOnly and routeByLatency options to scaffold.go
2019-12-20 19:27:15 -07:00
Dakota Sullivan
a0c9d9c0e7
Fix config property names
2019-12-20 19:23:01 -07:00
Chris Lu
71d55ec5cb
adjust help message
2019-12-20 08:22:32 -08:00
Chris Lu
d0aa0c5086
shell: add fs.meta.cat to see file meta data content
...
related to https://github.com/chrislusf/seaweedfs/issues/1163
2019-12-20 08:22:18 -08:00
Chris Lu
ab966d7192
refactoring variable name
2019-12-19 00:44:46 -08:00
Chris Lu
f61de28c69
volume: add deletion capability for previously readonly volumes
2019-12-19 00:42:46 -08:00
Chris Lu
deb03ecfa3
remove duplicated checking
...
already checked in store.go
2019-12-18 23:59:53 -08:00
Chris Lu
36ddca9d1f
filer: return not found 404 status if not found
...
fix https://github.com/chrislusf/seaweedfs/issues/1160
2019-12-18 21:04:40 -08:00
Chris Lu
ef3ae3cd41
clean up sorted index file
2019-12-18 20:59:15 -08:00
Dakota Sullivan
301bc2f69d
Add readOnly and routeByLatency options to Redis Cluster Store
2019-12-18 08:36:39 -07:00
Chris Lu
39810a9951
tiered volume is read only
2019-12-18 01:26:22 -08:00
Chris Lu
58f88e530c
volume: use sorted index map for readonly volumes
2019-12-18 01:21:21 -08:00
Chris Lu
2f21beaccd
loading xattr value
2019-12-17 21:45:32 -08:00
Chris Lu
014487b1a6
passing entry.Extended
2019-12-17 21:39:48 -08:00
Chris Lu
e707211dad
passing entry.exended
2019-12-17 21:22:16 -08:00
Chris Lu
29eb333f8e
fix comparing
2019-12-17 21:15:28 -08:00
Chris Lu
14242e6c70
passing xattr
2019-12-17 21:10:26 -08:00
Chris Lu
c42b023960
mount: invalidate xattr
2019-12-17 20:38:56 -08:00
Chris Lu
54369eccea
1.47
2019-12-17 15:11:01 -08:00
Chris Lu
6808bfec49
1.47
2019-12-17 15:08:45 -08:00
Chris Lu
9c781da34a
mount: fix nil directory problem
2019-12-17 15:08:38 -08:00
Chris Lu
ee6952fddd
twaking
2019-12-17 14:03:57 -08:00
Chris Lu
f27a1abf1c
mount: fix dir nil
2019-12-17 13:50:32 -08:00
bs106
ce79a31448
Fix missing leading dot in extension
2019-12-17 09:47:12 +00:00
Chris Lu
5961e0d9eb
fix logging
2019-12-15 22:24:06 -08:00
Chris Lu
33b4b1868b
mount: support extended attributes
2019-12-15 21:07:01 -08:00
Chris Lu
ad9d5ff5d0
adjust UI
2019-12-15 20:57:17 -08:00
Chris Lu
392678f8f3
upload skipping mimetype if not needed
2019-12-15 20:57:08 -08:00
Chris Lu
e6ee421d61
fix compilation
2019-12-14 12:04:25 -08:00
Chris Lu
a631b2a9e9
1.46
2019-12-14 11:28:48 -08:00
Chris Lu
33da30d2b1
tweak
2019-12-14 11:04:20 -08:00
Chris Lu
de3eb7c380
update cache if updating the attributes
2019-12-13 11:14:09 -08:00
Chris Lu
357ea4b944
rm will clean the cache first
2019-12-13 11:08:26 -08:00
Chris Lu
deb249736e
fix nill
2019-12-13 10:35:23 -08:00
Chris Lu
f59119cd1d
mount: rename to dirListCacheLimit to configure cache for long format directory list
2019-12-13 10:22:24 -08:00
Chris Lu
3e0cd122d8
mount: directory listing cache ttl set to 5 minutes
2019-12-13 10:08:27 -08:00
Chris Lu
05c3b795dc
filer: configurable directory list cache size
2019-12-13 10:05:43 -08:00
Chris Lu
bbb6ebc3c0
filer: DeleteFolderChildren for deleting large folders
2019-12-13 00:23:05 -08:00
Chris Lu
0fa1269bc7
filer: streaming file listing
2019-12-13 00:22:37 -08:00
Chris Lu
d0b423bbc0
filer: increase directory listing pagination size
2019-12-12 09:11:10 -08:00
Chris Lu
bf4b13612d
adjust command name
2019-12-11 23:13:04 -08:00
Chris Lu
1f400489c6
filer: UI ensure a correct foler link
2019-12-11 23:09:01 -08:00
Chris Lu
4cba32d1d0
go fmt
2019-12-11 22:22:09 -08:00
Chris Lu
f51a6d8ba5
increase filer pagination limit
2019-12-11 22:21:37 -08:00
Chris Lu
c602073395
adjust logs
2019-12-11 21:40:33 -08:00
Chris Lu
1ddfbff2d3
deletion add option skipChunkDeletion
2019-12-11 14:58:22 -08:00
Chris Lu
f06b65a6fd
filer: EXPERT add shouldDeleteChunks parameter for fast deletion
2019-12-11 07:50:53 -08:00
Yoni Nakache
9e45702334
Filer: Enable delete entries without flag volume
...
same as happens on the s3 API on delete bucket,
to reduce the IO requests on the system, you can just clear recursively folders without flagging them in volume server (they will be removed on delete bucket/drop collection)
this will remove the user ability to retrieve data under these folders
2019-12-11 11:30:19 +02:00
Chris Lu
40ff4eea14
fix nil problem
2019-12-10 23:13:14 -08:00
Chris Lu
356bd1b629
adjust text
2019-12-10 22:09:29 -08:00
Chris Lu
5847c5afff
return meaningful error log
2019-12-10 22:09:29 -08:00
Chris Lu
3727d2488f
Merge pull request #1149 from song-zhang/improve-replications-consistency
...
improve data consistency when replication write
2019-12-09 20:13:03 -08:00
Chris Lu
10bd3c6b4b
refactoring
2019-12-08 19:44:16 -08:00
Chris Lu
2b8e20f122
remove unused functions
2019-12-08 18:33:25 -08:00
zhangsong
5eb86aad47
improve data consistency when replication write
2019-12-08 22:21:35 +08:00
Chris Lu
e2a8cdfc48
webdav: fix writing file
...
fix https://github.com/chrislusf/seaweedfs/issues/1114
2019-12-07 07:56:51 -08:00
Chris Lu
0eda75fa2c
add logging
2019-12-07 07:56:05 -08:00
Chris Lu
f81d43442b
filer: speed up filer.meta.save by parallelizing
2019-12-06 22:20:59 -08:00
Chris Lu
f38f90b7ea
cleanly remove .tier volume file
2019-12-06 22:19:47 -08:00
Chris Lu
6827cabfbb
volume: dataFileAccessLock change to RW Lock
2019-12-06 06:59:57 -08:00
Chris Lu
81f424a630
go fmt
2019-12-06 06:57:32 -08:00
Chris Lu
916423e96a
master: submit support "dataCenter" parameter
2019-12-05 15:51:24 -08:00
Chris Lu
e426bd541e
grow volumes on volume servers with slots freed by cloud storage
2019-12-03 21:36:42 -08:00
Chris Lu
7ae8b1cc86
show volume info in volume.list in weed shell
2019-12-02 23:38:56 -08:00
Chris Lu
1becbce657
display remote volumes on volume server ui page
2019-12-02 23:23:54 -08:00
Chris Lu
70648d35ad
go fmt
2019-12-02 20:49:58 -08:00
Chris Lu
6383b45bd0
add lock variable
2019-12-02 20:49:50 -08:00
Chris Lu
caae543a9f
fix test
2019-12-02 15:54:24 -08:00
Chris Lu
126fb492d2
fix test
2019-12-02 15:52:33 -08:00
Chris Lu
ec8de250e2
tiered storage: can copy to s3, read from s3
...
master not aware tiered volume yet, file assigning is not working yet
2019-12-02 15:08:31 -08:00
Chris Lu
228231f3d7
minor
2019-12-02 15:08:31 -08:00
zhangsong
e83c36e26f
fix the bug of volume never be vacuumed
2019-12-02 13:25:32 +08:00
Chris Lu
0da7b894cc
pass backend config from master to volume servers
2019-11-29 01:05:09 -08:00
Chris Lu
a9dbd52e06
Merge pull request #1145 from stlpmo-jn/volume_graceful_stop
...
let volume server graceful stop
2019-11-29 00:11:57 -08:00
stlpmo
ea9d1ebd2f
resolve the Duration.Milliseconds() is not exist in golang v11 & v12
2019-11-29 15:17:17 +08:00
Chris Lu
ea9a7e61d0
only send metrics address during on the first response
2019-11-28 20:48:08 -08:00
Chris Lu
61bc1d6ffd
tweaking
2019-11-28 18:47:51 -08:00
Chris Lu
09c6d8fa32
fix test
2019-11-28 18:44:27 -08:00
Chris Lu
f60154f330
master load backend storage config from master.toml
2019-11-28 18:33:18 -08:00
Chris Lu
641b92f53c
fix scaffold error
2019-11-28 18:04:25 -08:00
Chris Lu
ee2072dff9
volume: avoid inifinite loop reading ec volume info if error happens
2019-11-27 23:07:53 -08:00
Chris Lu
0b822a6fb5
Merge pull request #1142 from iliul/wip-fuse-mount
...
fuse: check mount point available before do runmount
2019-11-27 19:22:51 -08:00
Lei Liu
13b6f09fea
fuse: adjust linux specific logic
...
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-11-28 10:21:31 +08:00
Chris Lu
586798ecc0
go fmt
2019-11-27 12:34:57 -08:00
Chris Lu
46294b42e6
filer: support memsql
...
fix https://github.com/chrislusf/seaweedfs/issues/1141
2019-11-27 12:34:03 -08:00
Lei Liu
1ba3456ac2
go fmt
...
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-11-27 23:08:51 +08:00
Lei Liu
4c87b222f1
fuse: check mount point available before do runmount
...
1. Use more readable mount point information
2. Fix some typos
eg:
$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
localhost:8888:/ fuse.seaweedfs 206G 512 206G 1% /mnt/weedfs
$ mount | grep weedfs
localhost:8888:/ on /mnt/weedfs type fuse.seaweedfs (rw,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-11-27 22:56:44 +08:00
Chris Lu
0f9ba84274
s3
2019-11-27 03:09:45 -08:00
Chris Lu
67e5f5b558
typo
2019-11-27 03:09:45 -08:00
Lei Liu
2700ac6d1f
filer: Add attr.mode in query string
...
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-11-27 13:57:29 +08:00
Chris Lu
5f6b360eb7
adjust error message
2019-11-24 23:13:40 -08:00
Chris Lu
2cbe79acda
fix typo in command help
2019-11-24 22:17:43 -08:00
Chris Lu
0fca1ae776
adjust build
2019-11-22 23:19:49 -08:00
Chris Lu
bc6cd93431
skip build for 32 bit computers
2019-11-22 22:52:21 -08:00
Chris Lu
249b9d8d5d
1.45
2019-11-22 14:51:16 -08:00
Chris Lu
e4c63ff2f8
go fmt
2019-11-22 14:51:09 -08:00
Chris Lu
fb3c9ed80d
shell: fs.meta.save missed directory data with more than 1024 entries
...
fix https://github.com/chrislusf/seaweedfs/issues/1130
2019-11-22 01:39:50 -08:00
divinerapier
5656d43264
can not break out of for-select block
...
Signed-off-by: divinerapier <poriter.coco@gmail.com>
2019-11-20 08:25:29 +08:00
Chris Lu
9711a6ffaa
WIP
2019-11-18 19:24:37 -08:00
Chris Lu
ba73c053c3
change uuid package
...
fix https://github.com/chrislusf/seaweedfs/issues/1126
2019-11-18 19:24:37 -08:00
chenwanli
295f3beed5
rm ldb and bdb directory
2019-11-19 09:35:06 +08:00
stlpmo
753138a3f7
test passed
2019-11-17 11:40:36 +08:00