Chris Lu
7e91ae592c
pass in option to read deleted entries
...
not working yet
2020-08-18 17:37:26 -07:00
Chris Lu
6a92f0bc7a
refactoring to typed Size
...
Go is amazing with refactoring!
2020-08-18 17:04:28 -07:00
Chris Lu
be4d42b8e2
rename
2020-08-17 20:15:53 -07:00
Chris Lu
aec7f32b02
fix reader_at
2020-08-16 00:49:08 -07:00
Chris Lu
c647deace1
file size support set file length
...
use Attr.FileSize and TotalChunkSize to determine file size
2020-08-15 09:32:47 -07:00
Chris Lu
3b1a95ac26
filer refactoring: same auto chunking logic for POST and PUT, no size limit
2020-08-08 12:02:06 -07:00
Chris Lu
67348e7b15
less noisy heartbeat logs
2020-08-08 10:53:35 -07:00
Chris Lu
bee0d7e5eb
lower log priority for noisy heartbeat
2020-08-08 10:52:13 -07:00
Chris Lu
ae00cce4bd
support POST and PUT auto chunking
2020-08-08 10:45:37 -07:00
Chris Lu
bd8bfdae07
refactoring
2020-08-08 10:18:43 -07:00
Chris Lu
20e2ac1add
filer: store md5 metadata for files uploaded by filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1412
2020-08-06 10:04:17 -07:00
Chris Lu
93ea0801ea
volume: the variable for the master node may be stale?
...
related to https://github.com/chrislusf/seaweedfs/issues/1414
2020-08-06 09:48:54 -07:00
Chris Lu
4ecfa9879d
volume: report Content-MD5 in response header
2020-08-06 05:22:53 -07:00
Устюжанин Антон Александрович
2d3b355fb6
test ListDirectoryPrefixedEntries
2020-08-05 23:56:06 +05:00
Устюжанин Антон Александрович
33a9e5e2d1
test ListDirectoryPrefixedEntries
2020-08-05 22:19:16 +05:00
Chris Lu
49199c69d4
volume: avoid special logic for .gz files
2020-08-01 11:19:57 -07:00
Chris Lu
3faa0aa448
rename
2020-07-25 20:09:19 -07:00
Chris Lu
023a1efdf2
set filename in Content-Disposition header
2020-07-25 20:06:40 -07:00
James Hartig
229f11c660
Added VolumeNeedleStatus volume server grpc method
...
This is needed for the diffing tool to get the cookie for a needle
2020-07-22 15:02:21 -04:00
Chris Lu
97d97f3528
go code can read and write chunk manifest
2020-07-19 17:59:43 -07:00
李远军
4162766c46
Fix the bug of " http://xx/xx,xxx.PNG?width=100 "
...
If you want to do a resize a UPPER ext pic file, there is a bug caused by commit 308688c8d0
(also see https://github.com/chrislusf/seaweedfs/blob/master/weed/images/resizing.go#L43 )
2020-07-17 19:57:35 +08:00
Chris Lu
ddec7b2bb9
go fmt
2020-07-14 07:34:16 -07:00
Chris Lu
4f6096c7f0
add reading from persisted logs for local filer store
2020-07-13 22:55:28 -07:00
Chris Lu
1dd2c76532
refactoring
2020-07-12 17:31:24 -07:00
Chris Lu
e7c04af1d0
skip http.StatusPartialContent in case of error and superfluous response.WriteHeader
2020-07-10 08:56:23 -07:00
李远军
6608cb5f43
Revert "remove fixJpgOrientation"
...
This reverts commit de5ca9b2
2020-07-10 10:08:36 +08:00
Chris Lu
a2eb680f34
filer: default filer store directory
...
1. set default filer store directory
2. set peers, avoiding empty string counted as 1.
2020-07-07 23:06:48 -07:00
Chris Lu
3c269da37f
fix compilation
2020-07-06 09:46:24 -07:00
Chris Lu
d198e8c6d6
filer: support cross filer meta data sync if sharing the same store
2020-07-06 09:24:54 -07:00
Chris Lu
0b8cc9b626
aggregate multiple filer metadata chagne events
2020-07-05 23:05:02 -07:00
Chris Lu
49929e0869
rename
2020-07-05 15:52:36 -07:00
Chris Lu
70d8a3a1d3
add SubscribeLocalMetadata without checking persisted meta logs
2020-07-05 15:50:07 -07:00
Chris Lu
55e40b08fc
refactoring
2020-07-05 15:43:06 -07:00
Chris Lu
881e0fde2e
rename
2020-07-05 08:55:09 -07:00
Chris Lu
4d1484628a
refactoring
2020-07-03 16:41:30 -07:00
Chris Lu
31e23e9783
filer: support active<=>active filer replication
2020-06-30 22:53:57 -07:00
Chris Lu
df75b5c98d
passing ctx for metadata logging
2020-06-28 14:34:51 -07:00
Chris Lu
a4dc25cd22
volume: generate ec shard first, before generating the ec index file.
...
possible fix https://github.com/chrislusf/seaweedfs/issues/1364
the ec index file is used to determine whether the ec files are ready or not
2020-06-25 11:06:58 -07:00
Chris Lu
3b638d3994
add more ec encoding logging
2020-06-25 09:43:38 -07:00
Chris Lu
fe60db404a
prepare for zstd
2020-06-23 09:12:02 -07:00
Chris Lu
a4e3cffe0a
S3: DeleteObject API also support deleting a folder recursively
2020-06-22 10:01:00 -07:00
Chris Lu
de5ca9b258
remove fixJpgOrientation
2020-06-20 12:50:40 -07:00
Chris Lu
3080c197e3
rename UnCompressData to DecompressData
2020-06-20 08:16:16 -07:00
Chris Lu
ca3516ac6d
adjust protoc
2020-06-20 08:00:25 -07:00
Chris Lu
e912fd15e3
renaming
2020-06-19 22:45:27 -07:00
Chris Lu
16fe132a20
detect mime type before replicating to other volume servers
2020-06-19 22:11:36 -07:00
Chris Lu
044841c885
master: always clear previous master meta data directory
2020-06-19 20:42:16 -07:00
Chris Lu
ae1994cbc1
erasure coding: fix cases where there are no .ecj files
2020-06-18 09:52:35 -07:00
Chris Lu
c66f357715
report error correctly
2020-06-10 10:59:25 -07:00
Chris Lu
c38066be89
also add it for webdav
2020-06-09 18:07:03 -07:00
Evgenii Kozlov
2410ffbd79
Name fix
2020-06-05 19:00:38 +03:00
Evgenii Kozlov
0e0db70f55
Set volumes ReadOnly if low free disk space
2020-06-05 18:18:15 +03:00
Chris Lu
393ab07c7c
disable pulse seconds
...
there are a few other hard coded timing settings, which will cause disconnected volume servers.
2020-06-04 10:52:01 -07:00
Chris Lu
bc2ec6774d
inject git version into build
2020-06-02 00:10:38 -07:00
bingoohuang
1a642b9876
add Volume Ids column only for max 100 volumes for convenience in the master ui.
2020-05-29 15:37:58 +08:00
bingoohuang
accb4964b7
util.BytesToHumanReadable
2020-05-29 10:00:07 +08:00
bingoohuang
21d0a013d5
rename 'needle' which collides with imported package name
2020-05-28 17:00:00 +08:00
bingoohuang
41cab5d69f
log JSON response if httpStatus >= 400
2020-05-28 08:56:55 +08:00
Chris Lu
d4235afe4d
support multiple locks
2020-05-25 18:39:53 -07:00
bingoohuang
ecdeef8c66
simplify func(w http.ResponseWriter, r *http.Request) to http.HandlerFunc
2020-05-25 21:00:12 +08:00
Chris Lu
91b91d6cb7
add error to avoid copying not found volume
...
fix https://github.com/chrislusf/seaweedfs/issues/1317
2020-05-17 20:20:12 -07:00
Chris Lu
3fbf635308
Merge branch 'master' into msg_channel
2020-05-13 03:54:15 -07:00
Chris Lu
3a5e4769cf
prevent empty locations returned
...
fix https://github.com/chrislusf/seaweedfs/issues/1313
2020-05-13 03:46:38 -07:00
Chris Lu
dfccc3c263
able to read chan and write chan
2020-05-08 02:47:22 -07:00
Chris Lu
1e3e4b3072
add broker connects to filer
2020-05-05 02:05:28 -07:00
Chris Lu
871efa4fc1
refactoring
...
some previous chunk etag was using md5, which should be wrong.
2020-04-30 17:20:47 -07:00
Chris Lu
9e72e9e4b8
able to subscribe any topic from any point of time
2020-04-29 17:40:08 -07:00
Chris Lu
ed3cf811f5
refactoring
2020-04-29 13:26:02 -07:00
Chris Lu
8d42a1263b
meta subscription: update the last read time
2020-04-28 17:30:04 -07:00
Chris Lu
31583b5d6c
master and volume server: avoid race condition
...
The volume server may disconnect and reconnect to the same master.
The master's unregistration may happen after the reconnection. Thus the volume server will disappear.
2020-04-28 17:29:10 -07:00
Chris Lu
98edec7c3e
remove printouts
2020-04-28 16:45:48 -07:00
Chris Lu
fb81f12686
filer: simplify image resize
2020-04-28 00:05:57 -07:00
Chris Lu
5c57297bd1
metadata log: read from any timestamp
2020-04-28 00:05:57 -07:00
wuyuxiang
6850d28d6b
refacotr: move signal handling and pprof to grace package
2020-04-28 14:10:23 +08:00
Chris Lu
b52b8ec685
Hadoop: fix entry not found for HCFS
...
also fix cipher related changes.
2020-04-26 05:21:54 -07:00
Chris Lu
662b5d0cf7
support semicolon seperated command lines
2020-04-23 14:01:46 -07:00
Chris Lu
73564e6a01
master: add cluster wide lock/unlock operation in weed shell
...
fix https://github.com/chrislusf/seaweedfs/issues/1286
2020-04-23 13:37:31 -07:00
Chris Lu
bdc337a719
make lock/unlock optional
2020-04-23 03:50:05 -07:00
Chris Lu
842e3301ee
make second lock instant
2020-04-23 03:32:18 -07:00
Chris Lu
30ee4f3291
add exclusive lock library on shell
2020-04-23 02:31:04 -07:00
Chris Lu
77873b832b
add master side code for cluster wise exclusive lock
2020-04-23 01:55:44 -07:00
Chris Lu
69eda558ef
reduce the connections
...
fix https://github.com/chrislusf/seaweedfs/issues/1277
this avoids sending error to wdclients, and clients retry the connections
2020-04-22 20:43:34 -07:00
Chris Lu
e24b25de78
async meta caching: can stream updates now
2020-04-21 21:16:13 -07:00
Chris Lu
27128c7875
adjust logging
2020-04-21 16:13:13 -07:00
Chris Lu
cb3985be70
go fmt
2020-04-20 17:48:06 -07:00
Chris Lu
ebfab42a50
refactoring
2020-04-20 00:08:47 -07:00
Chris Lu
2955b96ef1
refactor
2020-04-19 23:54:32 -07:00
Chris Lu
f373232227
Merge pull request #1280 from bukton123/filer_mongodb
...
Add new filer support mongodb
2020-04-19 20:35:19 -07:00
Chris Lu
076c8bd3bc
filer master start up with default ip address instead of just localhost
2020-04-18 15:17:27 -07:00
bukton
290c6b7f01
Merge remote-tracking branch 'origin/master' into filer_mongodb
...
# Conflicts:
# go.mod
# go.sum
# weed/server/filer_server.go
2020-04-19 00:21:45 +07:00
bukton
cd9cccec55
add import mongodb to filer server
2020-04-18 22:30:13 +07:00
Chris Lu
826f96b6f3
fix unreachable code
2020-04-17 10:04:28 -07:00
Chris Lu
bda82f61bc
filer: able to append to a file
2020-04-17 02:28:09 -07:00
Chris Lu
f5a748d33c
refactoring
2020-04-16 02:55:09 -07:00
Chris Lu
ce4b369be2
scaffolding messaging
2020-04-16 02:21:23 -07:00
Reed
aada5b9aa7
making volume stats size info readable.
2020-04-16 10:08:08 +08:00
Chris Lu
2b5c4fbbf3
tiered caching
...
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
2020-04-13 22:19:27 -07:00
Chris Lu
7764e0465c
refactoring
2020-04-12 21:00:55 -07:00
Chris Lu
8e23dc078b
refactoring
2020-04-12 20:48:21 -07:00
Chris Lu
5361f999ed
shell: only one shell is allowed to connect to the cluster
...
fix https://github.com/chrislusf/seaweedfs/issues/1266
2020-04-12 17:48:39 -07:00
Chris Lu
d30483d642
re-enable system logs
2020-04-12 14:03:07 -07:00
Chris Lu
6f948e4887
remove configurable topics folder location
2020-04-12 13:07:59 -07:00
Chris Lu
b5a713be68
filer store: add redis2 redis_cluster2 as default redis store
2020-04-12 02:50:41 -07:00
Chris Lu
1c65656fb4
s3: add option to fsync buckets
2020-04-11 23:37:10 -07:00
Chris Lu
e4af63a721
volume server: accept fsync=true in write requests
2020-04-11 21:39:16 -07:00
Chris Lu
df97da25f9
mount: add on disk caching
2020-04-11 21:12:41 -07:00
Chris Lu
d7f3acb2c0
refactor
2020-04-11 12:45:24 -07:00
Chris Lu
e909b55633
refactoring
2020-04-11 12:39:46 -07:00
Chris Lu
1101a42e5c
mv : create new folder, move children, and delete old folder
2020-04-10 01:37:03 -07:00
Chris Lu
f6a7e79dc3
weed shell: simplify CLI option for filer
2020-04-08 23:57:15 -07:00
Chris Lu
dc08e4098f
add etag only for PUT or large chunked uploads
2020-04-08 09:13:26 -07:00
Chris Lu
ec2eb8bc48
add If-None-Match and If-Modified-Since
...
fix https://github.com/chrislusf/seaweedfs/issues/1269
2020-04-08 08:12:00 -07:00
Chris Lu
745f5d2a25
fix setting default value
2020-04-07 01:58:48 -07:00
Chris Lu
54debdc6f7
filer: fix configuration settings
2020-04-07 01:30:53 -07:00
Chris Lu
f14b6a09a8
remove duplicated update events
2020-04-05 13:22:15 -07:00
Chris Lu
91da7057b1
refactoring
2020-04-05 13:11:43 -07:00
Chris Lu
2d43f85577
watch entries with common path prefix
2020-04-05 12:51:21 -07:00
Chris Lu
7bc3c93512
add util.PathJoin
2020-04-05 12:40:46 -07:00
Chris Lu
bf270d9e8c
filer: able to tail meta data changes
2020-04-05 00:51:16 -07:00
Chris Lu
6e776aaafc
loop through all masters
2020-04-03 00:47:48 -07:00
Chris Lu
4a5e381feb
fail fast if filer configured wrong masters
...
fix https://github.com/chrislusf/seaweedfs/issues/1257
2020-04-03 00:47:33 -07:00
Chris Lu
aaf5e658cc
adjust parameter name
2020-04-03 00:41:05 -07:00
Chris Lu
6c96e51414
re-ordering
2020-04-03 00:40:54 -07:00
Chris Lu
fee3319cbf
handle gzip for image resizing
2020-04-02 23:44:24 -07:00
Chris Lu
73bc286377
volume: passing ext info via URL
2020-04-02 21:23:15 -07:00
Chris Lu
cea52a4faf
volume copying adds cleaning up on error
...
fix https://github.com/chrislusf/seaweedfs/issues/1253
2020-04-02 15:36:55 -07:00
Chris Lu
c446438ca5
Merge pull request #1255 from levenlabs/ignore
...
Added treat_replication_as_minimums master toml option
2020-04-01 16:48:58 -07:00
James Hartig
eae3f27c80
Added treat_replication_as_minimums master toml option
2020-04-01 19:08:48 -04:00
Chris Lu
50a5018b7f
writing meta logs is working
2020-03-30 01:19:33 -07:00
Chris Lu
9dc0b1df8f
refactoring to compile for windows OS
...
windows os does not like to work with fuse
2020-03-29 21:07:55 -07:00
Chris Lu
a75d50bbb8
FUSE: add configurable in memory chunk cache size
2020-03-28 14:07:16 -07:00
Chris Lu
826bc0b7e3
FUSE: add chunk cache for recently accessed file chunks
2020-03-28 13:43:31 -07:00
Chris Lu
eedd33dda3
refactoring
2020-03-28 13:41:58 -07:00
Chris Lu
d1439c5bd3
fix FUSE read for large files
...
FUSE expects ReadAt do not return partial filled buffer with a nil error.
2020-03-27 04:50:51 -07:00
Chris Lu
34a0d424f8
FUSE mount: report max volume count available disk space
2020-03-26 23:06:21 -07:00
Chris Lu
a5ce359999
refactoring
2020-03-26 22:46:49 -07:00
Chris Lu
6d87a0c5f5
batch delete orphan data by file keys
2020-03-25 02:21:15 -07:00
Chris Lu
cddcb5cafe
same booleans as before
2020-03-23 01:25:23 -07:00
Chris Lu
f84c288852
purge
2020-03-23 01:14:55 -07:00
Chris Lu
a5aa8be19b
Update webdav_server.go
2020-03-23 01:14:42 -07:00
Chris Lu
c0f0fdb3ba
refactoring
2020-03-23 00:01:34 -07:00
Chris Lu
c16dc57a58
fix max volume count reporting
2020-03-22 18:33:19 -07:00
Chris Lu
3137777d83
volume: automatically detect max volume count
2020-03-22 16:21:42 -07:00
Chris Lu
9f2c9b666b
adjust log level
2020-03-22 01:39:08 -07:00
Chris Lu
ae2ee379c0
consistent 64bit size
2020-03-22 01:37:46 -07:00
Chris Lu
e93e986727
webdav: stream read
2020-03-22 01:30:27 -07:00
Chris Lu
65d2ea9fb0
FUSE mount: stream read data with buffer
...
fix https://github.com/chrislusf/seaweedfs/issues/1244
2020-03-22 01:00:36 -07:00
Chris Lu
308688c8d0
filer: add back image resizing capability
2020-03-20 20:31:11 -07:00
Chris Lu
f251d03673
refactoring
2020-03-20 15:46:16 -07:00
Chris Lu
11372dd548
refactoring
2020-03-20 15:38:04 -07:00
Chris Lu
c4bea45099
S3 API: fix DeleteMultipleObjectsHandler
...
fix https://github.com/chrislusf/seaweedfs/issues/1241
2020-03-20 14:17:31 -07:00
Chris Lu
d848d08944
use public url also for deletions
2020-03-18 11:16:45 -07:00
Chris Lu
bec6ec7db6
go fmt
2020-03-17 10:01:55 -07:00
Chris Lu
c3cb6fa1d7
volume: compaction can cause readonly volumes
...
address https://github.com/chrislusf/seaweedfs/issues/1233
2020-03-17 09:43:57 -07:00
Chris Lu
560df51def
refactoring
2020-03-15 03:11:26 -07:00
Chris Lu
7b37178716
filer: close stores if interrupted
2020-03-14 20:30:26 -07:00
Chris Lu
e85da50ad4
remove tikv, move its support to "tikv" branch
2020-03-11 20:59:00 -07:00
Chris Lu
4237a813cc
skip printouts
2020-03-11 01:13:54 -07:00
Chris Lu
d214cefc2e
filer: list diretory pagination make up for the expired entries
2020-03-09 23:28:01 -07:00
Chris Lu
0871d2cff0
volume: fix memory leak during compaction
...
fix https://github.com/chrislusf/seaweedfs/issues/1222
2020-03-09 22:29:02 -07:00
Chris Lu
89eb05b50f
filer: support TTL for all filer stores
2020-03-09 01:02:01 -07:00
Chris Lu
8a899992f2
filer: fix ttl parsing
...
fix https://github.com/chrislusf/seaweedfs/issues/1225
2020-03-09 00:16:10 -07:00
Chris Lu
2e3f6ad3a9
filer: remember content is gzipped or not
2020-03-08 21:39:33 -07:00
Chris Lu
e04c1a1361
filer: remove chunk manifest file support
...
This is not needed for filer
2020-03-08 17:03:27 -07:00
Chris Lu
9b3109a5d8
filer: processing all response headers, no pass through to volume server
...
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
2020-03-08 15:42:44 -07:00
Chris Lu
11fceaf2f7
refactoring
2020-03-07 18:09:30 -08:00
Chris Lu
1a75fbfce0
refactoring
2020-03-07 18:06:48 -08:00
Chris Lu
afb20de14c
breaks dependency loop
2020-03-07 17:01:39 -08:00
Chris Lu
9228ff192c
encryption works
2020-03-07 11:08:57 -08:00
Chris Lu
0375ce2c2e
filer: set mime type from volume server
2020-03-07 07:25:15 -08:00
Chris Lu
dba35404e4
filer: HEAD response add content type
2020-03-07 06:08:08 -08:00
Chris Lu
ea1169dc80
filer cipher: single chunk http POST and PUT and read
2020-03-07 06:08:08 -08:00
Chris Lu
13e215ee5c
filer: option to encrypt data on volume server
2020-03-06 00:49:47 -08:00
Chris Lu
f90c43635d
refactoring
2020-03-04 00:39:47 -08:00
Chris Lu
9b7dae8037
filer: fix uploaded chunked file size reporting bug.
2020-03-02 20:27:14 -08:00
Chris Lu
dc40413847
fix error
2020-03-01 22:21:13 -08:00
Chris Lu
6a8484b4ae
master able to list all master clients by type
2020-03-01 22:13:47 -08:00
Chris Lu
555413d9fc
weed queue starts
2020-02-27 00:07:13 -08:00
Chris Lu
8c1da71402
remove dead code
2020-02-26 17:27:49 -08:00
Chris Lu
0156e2975a
mount: add mode to run external to SeaweedFS container cluster
2020-02-26 16:46:01 -08:00
Chris Lu
97ab8a1976
remove ctx if possible
2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9
avoid reusing context object
...
fix https://github.com/chrislusf/seaweedfs/issues/1182
2020-02-25 21:50:12 -08:00
Chris Lu
0841bedb15
move filer assign volume grpc errror to response
2020-02-25 17:15:09 -08:00
Chris Lu
bc38b72a20
s3: implemented DeleteMultipleObjects
2020-02-25 14:38:36 -08:00
Chris Lu
6ab7368ef2
filer: dynamically create bucket under /buckets folder
2020-02-24 22:28:45 -08:00
Chris Lu
5bcb44eda9
filer: move buckets folder to filer.toml since it should not be changed often
2020-02-24 14:42:57 -08:00
Chris Lu
d8dec2323b
s3: move buckets folder configuration to filer
2020-02-24 14:34:14 -08:00
Chris Lu
b06b7ca6e6
adjust UI
2020-02-23 22:01:03 -08:00
Yoni Nakache
0d60bb4427
display bug, header is Free but data was Used
2020-02-24 07:47:27 +02:00
Chris Lu
346a1cf0b9
Merge pull request #1207 from LazyDBA247-Anyvision/update-stats-and-status-volume-server
...
Update stats and status volume server
2020-02-23 18:05:25 -08:00
LazyDBA247-Anyvision
4ff513d64d
staus route: add DiskStatuses for disk in the volume server status
...
whem monitoring server, better to know the status of the disks & volumes in a single route.
2020-02-23 23:27:09 +02:00
Yoni Nakache
cd4c9a365b
DiskStats: adding Total & Percent Usage
...
making relevant data visible and readable.
2020-02-23 22:33:47 +02:00
Chris Lu
f55f49970f
move volume server stats to grpc
2020-02-21 21:45:03 -08:00
Chris Lu
9b6296e77a
volume: add grpc file read operation
...
This is added more for performance benchmarking.
2020-02-14 00:37:32 -08:00
Chris Lu
2a6db0fd43
volume: return 204 for unchanged file uploads
...
fix https://github.com/chrislusf/seaweedfs/issues/1196
2020-02-11 09:45:02 -08:00
Chris Lu
382ff218d3
filer: set file size, streaming chunk file uploading
...
fix https://github.com/chrislusf/seaweedfs/issues/1193
2020-02-03 17:04:06 -08:00
Chris Lu
40ae533fa3
shell: add volume.configure.replication to change replication for a volume
...
fix https://github.com/chrislusf/seaweedfs/issues/1192
2020-02-02 15:37:23 -08:00
Chris Lu
d335f04de6
support env variables to overwrite toml file
2020-01-29 09:09:55 -08:00
Chris Lu
72a64a5cf8
use the same context object in order to retry
2020-01-26 14:42:11 -08:00
Chris Lu
c48fc8b4de
grpc send error via response instead of grpc error
2020-01-25 09:17:19 -08:00
Chris Lu
d4cde5df49
return empty response if not found
2020-01-24 18:07:34 -08:00
Chris Lu
3cdb27fafc
adjust logging
2020-01-24 17:55:39 -08:00
Chris Lu
e6f9f5da3a
adjust logging
2020-01-24 17:26:03 -08:00
Chris Lu
d4e75a0d18
filer: option to create file only if it is new, O_EXCL
2020-01-22 11:42:40 -08:00
Chris Lu
9b01a99d9a
adjust logging
2020-01-21 22:45:50 -08:00
Chris Lu
1b0bfbaf59
refactoring
2020-01-19 23:59:46 -08:00
Chris Lu
4e731f1c8b
volume: copy volumes also include .vif file
2020-01-04 11:28:29 -08:00
Chris Lu
df636e4c14
remove .vif files
2020-01-03 12:46:42 -08:00
Chris Lu
3eafec4b29
volume: add option to limit file size
2020-01-03 00:37:26 -08:00
Chris Lu
86d4b18eb5
filer: optional recursive deletion
...
fix https://github.com/chrislusf/seaweedfs/issues/1176
2019-12-31 11:52:54 -08:00
Chris Lu
509f314350
keep .vif file for ec to normal volume conversion
2019-12-28 21:52:55 -08:00
Chris Lu
0cc8347a3a
go fmt
2019-12-28 21:13:10 -08:00
Chris Lu
8fbc0a9163
fix edge cases
2019-12-28 12:59:31 -08:00
Chris Lu
37b64a50b4
ec: generate and copy .vif file
2019-12-28 12:44:59 -08:00
Chris Lu
2000284435
rename volume tier info to volume info
2019-12-28 11:21:49 -08:00
Chris Lu
b3c83f9404
fix spelling
2019-12-28 11:16:10 -08:00
Chris Lu
48d28d3eb2
tier: support remote file attributes and remember the file extension
2019-12-25 21:37:24 -08:00
Chris Lu
d960b3474a
tier storage: support downloading the remote dat files
2019-12-25 09:53:13 -08:00
Chris Lu
7ba6be2312
volume ui add readonly status
2019-12-24 14:55:26 -08:00
stlpmo
f87ccbddb4
remove the http 304 response body
2019-12-24 17:56:50 +08:00
Chris Lu
8a2d529759
go fmt
2019-12-23 12:48:53 -08:00
Chris Lu
09ca936c78
shell: add ec.decode command
2019-12-23 12:48:20 -08:00
Chris Lu
36ddca9d1f
filer: return not found 404 status if not found
...
fix https://github.com/chrislusf/seaweedfs/issues/1160
2019-12-18 21:04:40 -08:00
Chris Lu
58f88e530c
volume: use sorted index map for readonly volumes
2019-12-18 01:21:21 -08:00
Chris Lu
014487b1a6
passing entry.Extended
2019-12-17 21:39:48 -08:00
Chris Lu
e707211dad
passing entry.exended
2019-12-17 21:22:16 -08:00
Chris Lu
ad9d5ff5d0
adjust UI
2019-12-15 20:57:17 -08:00
Chris Lu
392678f8f3
upload skipping mimetype if not needed
2019-12-15 20:57:08 -08:00
Chris Lu
bbb6ebc3c0
filer: DeleteFolderChildren for deleting large folders
2019-12-13 00:23:05 -08:00
Chris Lu
0fa1269bc7
filer: streaming file listing
2019-12-13 00:22:37 -08:00
Chris Lu
d0b423bbc0
filer: increase directory listing pagination size
2019-12-12 09:11:10 -08:00
Chris Lu
1f400489c6
filer: UI ensure a correct foler link
2019-12-11 23:09:01 -08:00
Chris Lu
f51a6d8ba5
increase filer pagination limit
2019-12-11 22:21:37 -08:00
Chris Lu
1ddfbff2d3
deletion add option skipChunkDeletion
2019-12-11 14:58:22 -08:00
Chris Lu
f06b65a6fd
filer: EXPERT add shouldDeleteChunks parameter for fast deletion
2019-12-11 07:50:53 -08:00
Yoni Nakache
9e45702334
Filer: Enable delete entries without flag volume
...
same as happens on the s3 API on delete bucket,
to reduce the IO requests on the system, you can just clear recursively folders without flagging them in volume server (they will be removed on delete bucket/drop collection)
this will remove the user ability to retrieve data under these folders
2019-12-11 11:30:19 +02:00
Chris Lu
10bd3c6b4b
refactoring
2019-12-08 19:44:16 -08:00
Chris Lu
e2a8cdfc48
webdav: fix writing file
...
fix https://github.com/chrislusf/seaweedfs/issues/1114
2019-12-07 07:56:51 -08:00
Chris Lu
0eda75fa2c
add logging
2019-12-07 07:56:05 -08:00
Chris Lu
81f424a630
go fmt
2019-12-06 06:57:32 -08:00
Chris Lu
916423e96a
master: submit support "dataCenter" parameter
2019-12-05 15:51:24 -08:00
Chris Lu
e426bd541e
grow volumes on volume servers with slots freed by cloud storage
2019-12-03 21:36:42 -08:00
Chris Lu
1becbce657
display remote volumes on volume server ui page
2019-12-02 23:23:54 -08:00
Chris Lu
6383b45bd0
add lock variable
2019-12-02 20:49:50 -08:00
Chris Lu
ec8de250e2
tiered storage: can copy to s3, read from s3
...
master not aware tiered volume yet, file assigning is not working yet
2019-12-02 15:08:31 -08:00
Chris Lu
0da7b894cc
pass backend config from master to volume servers
2019-11-29 01:05:09 -08:00
Chris Lu
ea9a7e61d0
only send metrics address during on the first response
2019-11-28 20:48:08 -08:00
Chris Lu
f60154f330
master load backend storage config from master.toml
2019-11-28 18:33:18 -08:00
Chris Lu
586798ecc0
go fmt
2019-11-27 12:34:57 -08:00
Chris Lu
0f9ba84274
s3
2019-11-27 03:09:45 -08:00
Lei Liu
2700ac6d1f
filer: Add attr.mode in query string
...
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-11-27 13:57:29 +08:00
Chris Lu
fb3c9ed80d
shell: fs.meta.save missed directory data with more than 1024 entries
...
fix https://github.com/chrislusf/seaweedfs/issues/1130
2019-11-22 01:39:50 -08:00
Chris Lu
de0c983808
refactoring scaffolding for master.toml
2019-11-14 22:20:38 -08:00
Lei Liu
46755ea1e1
fix master maintenance logic
...
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-11-12 14:46:10 +08:00
Chris Lu
ee90236a97
Merge branch 'master' into refactoring_dat_backend
2019-11-11 21:30:14 -08:00
Chris Lu
d7852ebb3b
Merge pull request #1102 from stlpmo-jn/add_ETCD_sequencer
...
Add etcd sequencer
2019-11-11 12:43:24 -08:00
stlpmo
62d393d6c9
ut pass
2019-11-11 10:52:21 +08:00
stlpmo
802a0eb3fe
move from cmd-line to scaffold
2019-11-11 09:15:17 +08:00
zhangsong
61fa485700
add volume number param in assign operation
2019-11-10 20:11:03 +08:00
Chris Lu
c5c1d83d91
Merge branch 'master' into refactoring_dat_backend
2019-11-08 22:54:41 -08:00
Chris Lu
6cc9e0d788
volume: fix ec shard reading
...
fix https://github.com/chrislusf/seaweedfs/issues/1106
2019-11-08 22:41:02 -08:00
stlpmo
1c8bed3810
delete the var etcdClient and comments
...
refactor the code
add sequencer cmd-line
delete nerver used codes
2019-11-05 15:45:25 +08:00
Lei Liu
1294999d8b
return error when garbageThreshold is not a valid float number
...
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-10-31 20:43:20 +08:00
Lei Liu
1bcef02a6c
fix dir/lookup and col/delete api
...
1, Fix Layouts first letter capitalized
2, Return http 204 when delete a collection
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-10-31 16:55:45 +08:00
Lei Liu
f2f90436ef
fix leader master /dir/lookup api
...
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-10-30 16:38:40 +08:00
Chris Lu
5b950c735e
Merge branch 'master' into refactoring_dat_backend
2019-10-29 23:18:41 -07:00
Lei Liu
c2884cace2
misc updated
...
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-10-29 21:28:28 +08:00
Chris Lu
19b6a16003
changed from os.file to backend.DataStorageBackend
2019-10-29 00:35:16 -07:00
Chris Lu
71eb8efd20
fix error logs
2019-10-25 09:44:58 -07:00
Chris Lu
55a2221eba
master: add logging for volume server disconnection
...
add logging for https://github.com/chrislusf/seaweedfs/issues/1093
2019-10-25 09:35:19 -07:00
Chris Lu
46ed2ca902
refactoring
2019-10-24 23:41:32 -07:00
divinerapier
f1ee6b7a54
fix abused 404 status code
...
Signed-off-by: divinerapier <poriter.coco@gmail.com>
2019-10-24 16:51:26 +08:00
Chris Lu
c358f96aaa
volume: redirection follow public url
...
fix https://github.com/chrislusf/seaweedfs/issues/1089
2019-10-23 22:11:42 -07:00
Chris Lu
23240efa63
filer: add tikv support
2019-10-23 00:31:31 -07:00
Chris Lu
c71fddee2b
go fmt
2019-10-22 00:50:50 -07:00
Chris Lu
fc412e428b
refactor ScanVolumeFileFrom()
2019-10-22 00:50:30 -07:00
Chris Lu
c9a183eb69
refactor memory map related code
2019-10-22 00:49:42 -07:00
Chris Lu
faec9076a4
adjust parameter names
2019-10-21 22:57:01 -07:00
Chris Lu
d16c450682
fix unreachable code
2019-10-21 22:36:41 -07:00
joeslay
d53aee179b
Merge pull request #10 from chrislusf/master
...
merge seaweed master
2019-10-14 16:03:40 +01:00
Chris Lu
939e4b57a8
go fmt
2019-10-09 00:03:18 -07:00
Chris Lu
f8d4b7d1c0
support basic json filtering and selection
2019-10-06 22:35:08 -07:00
Chris Lu
cf47f657af
scaffold for volume server query feature
2019-10-06 22:35:08 -07:00
joeslay
9d03aa6a38
Merge pull request #8 from chrislusf/master
...
merge seaweed master
2019-10-01 17:55:31 +01:00
Lei Liu
3cc084269c
master api: return http 404 when volumeId not exist
...
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-09-29 14:23:39 +08:00
j.laycock
6fc6322c90
Change joeslay paths to chrislusf paths
2019-09-12 14:18:21 +01:00
joeslay
d8c34b032f
Merge pull request #6 from chrislusf/master
...
merge seaweed master
2019-09-12 11:44:09 +01:00
Chris Lu
ae53f63680
filer: recursive deletion optionally ignoring any errors
...
fix https://github.com/chrislusf/seaweedfs/issues/1062
2019-09-11 20:26:20 -07:00
Tom Maxwell
4a878c0006
Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve
2019-09-04 15:27:14 +01:00
joeslay
6ee65356e3
Merge pull request #3 from chrislusf/master
...
merge seaweed master
2019-09-04 14:58:04 +01:00
j.laycock
cbd0a98fa1
Add InMemory to Volume Grow structure
2019-09-03 18:19:02 +01:00
Chris Lu
ee8bcb0c12
remove println
2019-09-03 09:22:56 -07:00
j.laycock
1f01eb78e8
Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct
2019-09-03 17:00:59 +01:00
Tom Maxwell
d637d86d22
Changes to try and pass the URL parameters through - in memory flag not working still
2019-09-03 15:41:28 +01:00
joeslay
e91139348e
Merge pull request #2 from chrislusf/master
...
merge seaweed master
2019-09-02 11:39:40 +01:00
j.laycock
595a1beff0
Swap imports to use joeslay
2019-09-02 11:28:40 +01:00
Chris Lu
60c9215a00
HCFS: avoid verbose file not found exception stack trace
2019-09-01 21:40:26 -07:00
Chris Lu
0dc43ecdca
skip writing content if not modified
...
related to https://github.com/chrislusf/seaweedfs/issues/1036
2019-08-09 14:38:54 -07:00
Feng Ye
fc0498798b
Add etcd filestore
2019-08-01 10:16:45 +08:00
Chris Lu
ca2b81a944
fix test error
2019-07-31 02:09:13 -07:00