Konstantin Lebedev
42f631f549
fix https://github.com/chrislusf/seaweedfs/issues/2046
2021-05-20 13:19:35 +05:00
Konstantin Lebedev
03d1199d5f
Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader"
...
This reverts commit ac71117e
2021-05-20 11:45:21 +05:00
Chris Lu
1737af480a
adjust logs
2021-05-10 21:47:51 -07:00
Chris Lu
55a8f57381
go fmt
2021-05-06 03:37:51 -07:00
Chris Lu
ac71117ee6
revert PR #1903 avoid http error: superfluous response.WriteHeader
2021-05-05 15:11:39 -07:00
Chris Lu
fa0d973113
Merge pull request #2037 from utsl42/reader_at
...
make reader_at handle random reads more efficiently for FUSE
2021-04-28 17:31:46 -07:00
Nathan Hawkins
042de9359c
make reader_at handle random reads more efficiently for FUSE
2021-04-28 19:13:37 -04:00
Konstantin Lebedev
c2269123d3
fix aws style Etag for chunks
2021-04-28 22:28:05 +05:00
Chris Lu
79f2e780c1
ensure name pattern checking is case sensitive
2021-04-24 11:51:23 -07:00
Chris Lu
ddc8643ee0
filer: directory listing adds namePatternExclude
...
fix https://github.com/chrislusf/seaweedfs/issues/2023
2021-04-24 11:49:03 -07:00
Chris Lu
89eb9f6e70
clean up .uploads directory in mysql and postgres tables
...
fix https://github.com/chrislusf/seaweedfs/issues/1957
When no uploads are running, you can run this SQL to clean up.
delete from <bucket_name> where directory like '/.uploads/%'
2021-04-22 23:23:23 -07:00
Chris Lu
d41e6826d3
adjust logging
2021-04-18 13:06:38 -07:00
Konstantin Lebedev
5021bea698
GetUserPolicy
2021-04-10 23:57:45 +05:00
Konstantin Lebedev
011e6e90ee
Merge branch 'upstreamMaster' into iamapipr
2021-04-06 13:50:33 +05:00
Chris Lu
5e64f65632
ensure tables are created
...
fix https://github.com/chrislusf/seaweedfs/issues/1957
2021-04-02 12:02:26 -07:00
Chris Lu
8e404a1433
go fmt
2021-04-02 02:22:26 -07:00
Chris Lu
e06807a55b
adjust logs
2021-03-30 20:36:06 -07:00
LazyDBA247-Anyvision
7f44d953b5
fix GetBool
2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision
4c51e6a660
add enableUpsert=true
...
and rename config to upsertQuery
2021-03-30 00:32:03 +03:00
LazyDBA247-Anyvision
4a02389eb0
Adding custom insertQuery support for postgres/2 mysql/2
2021-03-29 09:58:13 +03:00
Chris Lu
bd7471d877
refactor
2021-03-25 12:05:59 -07:00
Konstantin Lebedev
9f26f2815c
SaveAs S3 Configuration
2021-03-25 12:34:11 +05:00
Chris Lu
2f7c7afdec
refactor: remove unused parameter
2021-03-22 22:13:19 -07:00
Chris Lu
d48dd0c738
use []byte directly instead of bytes.Buffer
2021-03-22 22:12:57 -07:00
Konstantin Lebedev
eb54993a4e
Merge branch 'upstreamMaster' into check_chunkviews_mr
...
# Conflicts:
# weed/filer/filechunk_manifest.go
# weed/filer/stream.go
# weed/replication/repl_util/replication_util.go
# weed/util/fasthttp_util.go
2021-03-16 15:29:49 +05:00
Chris Lu
69694a17be
reverting 7d57664c2d
2021-03-16 02:59:26 -07:00
Konstantin Lebedev
06da02739d
CheckAllChunkViews() for HEAD requests only
2021-03-16 14:15:17 +05:00
Chris Lu
4b1ed227d1
revert fasthttp changes
...
related to https://github.com/chrislusf/seaweedfs/issues/1907
2021-03-16 00:33:14 -07:00
Konstantin Lebedev
46b9f5cff4
add debug logging
2021-03-15 19:33:33 +05:00
Konstantin Lebedev
7194a5e7bf
avoid http error: superfluous response.WriteHeader
...
https://github.com/chrislusf/seaweedfs/issues/1838
2021-03-15 18:52:59 +05:00
Chris Lu
35d939a1d2
go fmt
2021-03-14 13:21:02 -07:00
Chris Lu
2d4c2db81d
filer: leveldb, rocksdb auto create store directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1901
2021-03-14 13:20:14 -07:00
Chris Lu
6d3a96eb56
filer: mysql2, postgres2 trigger actions on bucket creation and deletion
...
fix https://github.com/chrislusf/seaweedfs/issues/1877
2021-03-13 22:07:39 -08:00
Chris Lu
7d57664c2d
mount: internals switch to filer.Entry instead of protobuf
2021-03-12 00:36:38 -08:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder
...
s3 delete-objects pass OK for fail to delete non-empty folder
2021-03-11 11:01:44 -08:00
Konstantin Lebedev
58cdcc6d6e
error if urls not found
2021-03-11 23:34:36 +05:00
Konstantin Lebedev
4db21012c1
error if read chunk zero data size
2021-03-11 22:38:59 +05:00
Konstantin Lebedev
15b018da34
s3 delete-objects pass OK for fail to delete non-empty folder
2021-03-10 19:41:35 +05:00
Chris Lu
5be83dd4a9
add comments
2021-03-09 14:13:48 -08:00
Chris Lu
9d8ca513d9
refactor
2021-03-09 13:21:26 -08:00
Chris Lu
9abb041763
filer source: support filerProxy mode
2021-02-28 16:19:47 -08:00
Chris Lu
776f497469
filer: fs.configure should try to read from entry.content also
...
related to https://github.com/chrislusf/seaweedfs/issues/1792
2021-02-18 17:07:02 -08:00
Chris Lu
3575d41009
go fmt
2021-02-17 20:57:08 -08:00
Chris Lu
6daa932f5c
refactoring to get master function, instead of passing master values directly
...
this will enable retrying later
2021-02-17 20:55:55 -08:00
Chris Lu
3f8b0da677
filer: do not print password on error
...
fix https://github.com/chrislusf/seaweedfs/issues/1809
2021-02-17 02:13:52 -08:00
Chris Lu
b9b5b932c5
filer: postgres2 mysql2 avoid repeatedly creating tables
2021-02-15 16:19:24 -08:00
LazyDBA247-Anyvision
7f458d5e78
better postgres connection pool management
...
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime )
to enable refresh of stale connections.
2021-02-15 07:45:09 +02:00
LazyDBA247-Anyvision
51b4963e2e
postgres2 & memsql2
...
add escape (quote identifiers) for the dynamic sql
so tables (collections) with special characters will work.
2021-02-14 13:14:36 +02:00
Chris Lu
7d9dc3c6a2
use fasthttp lib to read
2021-02-12 05:59:52 -08:00
Konstantin Lebedev
94eac4f00e
Do reconnect to RabbitMQ
...
https://github.com/google/go-cloud/issues/2958
https://github.com/chrislusf/seaweedfs/issues/1773
2021-02-11 13:59:36 +05:00
Chris Lu
821c46edf1
Merge branch 'master' into support_ssd_volume
2021-02-09 11:37:07 -08:00
Chris Lu
c3af72d950
fier store: fix elastic search regression
...
fix https://github.com/chrislusf/seaweedfs/issues/1774
2021-02-04 01:30:14 -08:00
Chris Lu
2396ac234c
filer: fix elastic search pagination
...
possible fix for https://github.com/chrislusf/seaweedfs/issues/1774
2021-02-02 11:34:12 -08:00
Chris Lu
4be51c0701
filer: leveldb and hbase may miss files when listing large directories more than 1024
...
fix https://github.com/chrislusf/seaweedfs/issues/1768
2021-01-31 20:11:44 -08:00
Chris Lu
314dc1c957
filer: etcd store fix listing
...
fix https://github.com/chrislusf/seaweedfs/issues/1767
2021-01-30 13:38:44 -08:00
Chris Lu
990fa69bfe
add back AdjustedUrl() related code
2021-01-28 14:36:29 -08:00
Chris Lu
00707ec00f
mount: outsideContainerClusterMode proxy through filer
...
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
2021-01-24 19:01:58 -08:00
Chris Lu
6ca10725b8
Revert "mount: when outside cluster network, use filer as proxy to access volume servers"
...
This reverts commit 096e088d7b
.
2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b
mount: when outside cluster network, use filer as proxy to access volume servers
2021-01-24 01:41:38 -08:00
Chris Lu
d5add83e85
filer store: add postgres2
2021-01-19 18:07:29 -08:00
Chris Lu
52a8f1470e
filer store: add mysql2
2021-01-19 17:21:50 -08:00
Chris Lu
93b3adba98
fix bucket creation
2021-01-19 15:55:51 -08:00
Chris Lu
fa0c8d5283
fix error
2021-01-19 14:05:48 -08:00
Chris Lu
4c5b752b04
restructuring sql stores
2021-01-19 13:53:16 -08:00
Chris Lu
ca8f793978
refactor: prepare for bucket specific tables
2021-01-19 12:34:58 -08:00
Chris Lu
90ce1eec19
add more input params
2021-01-19 11:32:55 -08:00
Chris Lu
de876c795d
minor fix
2021-01-18 01:14:27 -08:00
Chris Lu
09f49d1c04
refactoring
2021-01-16 19:52:15 -08:00
Chris Lu
a4063a5437
add stream list directory entries
2021-01-15 23:56:24 -08:00
Chris Lu
01dc8a43ba
fix rocksdb
2021-01-15 18:31:29 -08:00
Chris Lu
16ad74f477
go fmt
2021-01-14 23:11:27 -08:00
Chris Lu
f002e668de
change limit to int64 in case of overflow
2021-01-14 23:10:37 -08:00
Chris Lu
19e52fd414
pass along hasMore
2021-01-14 22:49:29 -08:00
Chris Lu
c0bcf6f6e1
implement sql changes
2021-01-14 22:44:22 -08:00
Chris Lu
5d4568b91f
implement elastic changes
2021-01-14 22:42:25 -08:00
Chris Lu
893cbc8482
implement c* changes
2021-01-14 22:38:34 -08:00
Chris Lu
5ef43b9b09
implement etcd changes
2021-01-14 22:35:56 -08:00
Chris Lu
54527f0326
implement hbase changes
2021-01-14 22:34:35 -08:00
Chris Lu
b5ceffe188
implement leveldb changes
2021-01-14 22:33:05 -08:00
Chris Lu
c64bfb0e2e
implement mongodb changes
2021-01-14 22:28:51 -08:00
Chris Lu
34a846009d
implement rocksdb changes
2021-01-14 22:24:26 -08:00
Chris Lu
d2ece29fff
implement redis changes
2021-01-14 22:21:31 -08:00
Chris Lu
9a50dbcda0
chagned api
2021-01-14 22:18:38 -08:00
Chris Lu
e1c7bc66f9
change interface
2021-01-14 22:17:49 -08:00
Chris Lu
4aaa97c789
skip not found error in case of race condition
2021-01-14 19:56:14 -08:00
Chris Lu
fd1d8a2a09
fix compilation
2021-01-14 00:06:15 -08:00
LazyDBA247-Anyvision
8eed763b97
better mysql connection pool management
...
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime )
to enable refresh of connections.
2021-01-14 08:14:21 +02:00
Chris Lu
f17aa1d06c
skip directory loop if dropping the bucket
2021-01-13 13:49:04 -08:00
Chris Lu
ca73013453
leveldb3: ensure repeatable adding removing buckets
2021-01-13 13:48:48 -08:00
Chris Lu
5cd9c9e642
remove println
2021-01-12 18:47:38 -08:00
Chris Lu
def6d8e565
fix test
2021-01-12 12:32:54 -08:00
Chris Lu
629c9962e7
filer: add leveldb3
...
support dedicated leveldb instance for each bucket
2021-01-12 02:29:44 -08:00
Chris Lu
361043e6c1
filer store: leveldb2 fix nil entry error if not found
2021-01-12 02:28:57 -08:00
Chris Lu
cfb9342a15
avoid concurrent map updates to viper
2021-01-12 02:28:13 -08:00
Chris Lu
acc0d5cd2c
Revert "Revert "filer store: redis module upgrade""
...
This reverts commit dd0171e35c
.
2021-01-11 02:30:19 -08:00
Chris Lu
dd0171e35c
Revert "filer store: redis module upgrade"
...
This reverts commit 40161c9cec
.
2021-01-11 02:09:18 -08:00
Chris Lu
394513f598
filer: ensure seamless meta data updates
2021-01-11 02:08:55 -08:00
Chris Lu
e2e0e30a51
filer: redis default to false for useReadOnly and routeByLatency
2021-01-08 01:12:44 -08:00
Chris Lu
40161c9cec
filer store: redis module upgrade
2021-01-08 01:08:44 -08:00
Chris Lu
07f4703bfc
Merge pull request #1738 from qieqieplus/rocksdb
...
fix rocksdb crash when list directory
2021-01-07 02:26:41 -08:00
qieqieplus
0f18592315
alter style
2021-01-07 18:21:48 +08:00
qieqieplus
c7e7431fd3
fix seek lastkey may reach EOF
2021-01-07 17:07:56 +08:00
Chris Lu
e327385644
go fmt
2021-01-06 04:22:00 -08:00
Chris Lu
2b76854641
add "weed filer.cat" to read files directly from volume servers
2021-01-06 04:22:00 -08:00
qieqieplus
0764fccde7
ignore decode error for non-entry data
2021-01-06 17:42:43 +08:00
qieqieplus
b2e50f602f
fix time comparison
2021-01-05 19:36:40 +08:00
qieqieplus
bbae13e415
impl: TTL per entry for rocksdb; fix package name
2021-01-05 17:58:40 +08:00
qieqieplus
ebb223c190
fix for rocksdb kv
2021-01-04 21:21:46 +08:00
chenqieqie
40b3207f18
fix #1726
2021-01-04 18:48:55 +08:00
Chris Lu
bdd166c031
adjust readme
2021-01-03 00:47:58 -08:00
Chris Lu
f4d87f8b9d
works
...
performance seems about the same as leveldb for insert entries
2021-01-02 23:32:58 -08:00
Chris Lu
2c3c2c27d7
separate prefix from namePattern
...
fix https://github.com/chrislusf/seaweedfs/issues/1722
2021-01-01 20:23:23 -08:00
Chris Lu
da7e5aaa65
filer: optimize for less number of directory lookup
...
bottom up directory lookup
2020-12-27 20:46:17 -08:00
Chris Lu
da87f6b265
remove unused code
2020-12-26 15:21:12 -08:00
Chris Lu
0a7c5f85a9
filer: add namePattern to search in current folder
2020-12-26 15:05:31 -08:00
Chris Lu
0e016bc7bd
hbase add ttl
2020-12-24 12:10:35 -08:00
Chris Lu
9bf6c10505
fix prefix
2020-12-24 00:18:59 -08:00
Chris Lu
2fea8cfa0f
fix compilation
2020-12-23 23:54:15 -08:00
Chris Lu
53bc1ea25b
fix compilation
2020-12-23 23:53:46 -08:00
Chris Lu
b5e2be635a
adjust for directory listing
2020-12-23 23:49:22 -08:00
Chris Lu
c4a202ec41
fix wrong column family
2020-12-23 23:23:05 -08:00
Chris Lu
94e3757c08
fix
2020-12-23 21:45:16 -08:00
Chris Lu
64c48c9724
seems compiling
2020-12-23 21:44:53 -08:00
Chris Lu
1c7e1295dc
Merge branch 'master' into support_ssd_volume
2020-12-22 17:44:52 -08:00
Chris Lu
a0990b929d
minor
2020-12-22 17:33:40 -08:00
Chris Lu
97e3432dfe
avoid wrong error
...
fix https://github.com/chrislusf/seaweedfs/issues/1691
2020-12-22 02:46:24 -08:00
Chris Lu
4f31c1bb94
go fmt
2020-12-22 02:34:08 -08:00
Chris Lu
6c4f32d173
remove unused code
2020-12-22 02:33:31 -08:00
Chris Lu
90d785a15f
filer: redis, redis cluster, cassandra support super large directory
2020-12-22 02:26:05 -08:00
Chris Lu
fe46411cd4
handle directory listing edge cases
2020-12-21 23:37:43 -08:00
Chris Lu
c74bede730
handle listing directories
2020-12-21 23:19:05 -08:00
Chris Lu
adf8cb4000
filer: path-specific stores trim out common prefixes
2020-12-21 22:57:13 -08:00
Chris Lu
0823bde8d8
refactor: split into two files
2020-12-21 22:05:15 -08:00
Chris Lu
8ab70ccf76
refactor
2020-12-21 21:46:32 -08:00
Chris Lu
65cc6dc636
refactor to dedicated function
2020-12-21 21:18:34 -08:00
Chris Lu
488c2680e8
change default filer store root
2020-12-21 21:15:22 -08:00
Chris Lu
f5a19a87be
filer: report error if the path specific store is not found
...
related https://github.com/chrislusf/seaweedfs/issues/1691
2020-12-21 12:43:52 -08:00
Chris Lu
41c0f3ad24
filer: support path-specific filer store
2020-12-19 01:27:09 -08:00
Chris Lu
0d5683fb0e
todo: load path-specific store from filer.toml
2020-12-18 03:05:42 -08:00
Chris Lu
3269fd7eaf
filer: use store by path
2020-12-18 02:57:49 -08:00
Chris Lu
e605f1e001
refactor, add getActualStore() by path
2020-12-18 02:55:00 -08:00
Chris Lu
23903aa95b
refactoring
2020-12-18 02:35:45 -08:00
Chris Lu
4e6b316913
assert FilerStoreWrapper is VirtualFilerStore
2020-12-18 02:17:06 -08:00
Chris Lu
1bf22c0b5b
go fmt
2020-12-16 09:14:05 -08:00
Chris Lu
c45d6a865b
Merge branch 'master' into support_ssd_volume
2020-12-13 22:14:18 -08:00
Chris Lu
38fc89d041
filer store: sql update if any insert error happens
...
fix https://github.com/chrislusf/seaweedfs/issues/1673
2020-12-13 20:49:44 -08:00
Chris Lu
23280257df
sql put kv: avoid unnecessary update
2020-12-13 19:45:47 -08:00
Chris Lu
0d2ec832e2
rename from volumeType to diskType
2020-12-13 11:59:32 -08:00
Chris Lu
e9cd798bd3
adding volume type
2020-12-13 00:58:58 -08:00
Chris Lu
14910d035c
the pagination size was too big for recursive deletion
2020-12-12 13:26:10 -08:00
Chris Lu
03637d6f57
s3: move "delete-directory-if-empty" to read time
...
move "delete-directory-if-empty" to read time instead of entry deletion time
the listing speed for a s3 bucket folder will slow down if it has many sub folders
related to 0d345ac97d
fix https://github.com/chrislusf/seaweedfs/issues/1647
fix https://github.com/chrislusf/seaweedfs/issues/1670
2020-12-12 03:38:34 -08:00
Chris Lu
37075a414d
adjust logs
2020-12-12 03:33:57 -08:00
Chris Lu
3fedfec1e7
check cross device rename error
2020-12-10 23:50:32 -08:00
Chris Lu
c2f18a10cb
minor
2020-12-10 22:23:22 -08:00
Chris Lu
4b0c2a846b
add logs
2020-12-10 19:55:28 -08:00
Chris Lu
83078ac6ce
filer: change to /etc/seaweedfs folder on filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1666
2020-12-10 11:11:02 -08:00
Chris Lu
765b3ef1be
save /etc/iam/identity.json inside filer store
2020-12-10 00:15:22 -08:00
Chris Lu
a9c619c270
added error handling for saving
2020-12-09 01:34:24 -08:00
Chris Lu
8e78187a97
add back last read chunk cache to reader and properly close the reader
2020-12-08 22:26:46 -08:00
Chris Lu
900d22c6ec
mount: avoid memory leaking read buffer
...
fix https://github.com/chrislusf/seaweedfs/issues/1654
the reader goes together with the file handle, which may stay for a long time.
2020-12-08 02:38:53 -08:00
Chris Lu
fce8803087
break import cycle
2020-12-07 00:29:17 -08:00
Chris Lu
01e2da5782
refactoring
2020-12-06 23:16:20 -08:00
Chris Lu
9acda432fe
fix import cycle
2020-12-06 20:12:52 -08:00
Chris Lu
5931a2f53f
filer conf: support json conf only
2020-12-06 19:47:06 -08:00
Chris Lu
f4abd01adf
filer: cache small file to filer store
2020-11-30 04:34:04 -08:00
Chris Lu
a9c6be5fc3
set creation time
2020-11-30 03:11:52 -08:00
Chris Lu
0d345ac97d
s3: remove empty parent folder on delete
...
fix https://github.com/chrislusf/seaweedfs/issues/1637
2020-11-26 12:21:58 -08:00
Chris Lu
cc2bd97ad9
refactor
2020-11-26 11:25:56 -08:00
Chris Lu
0da7ecfd29
go fmt
2020-11-26 11:22:30 -08:00
Chris Lu
effa00ed08
refactor
2020-11-26 11:21:58 -08:00
Chris Lu
513bcd6e0d
filer: avoid duplicated FindEntry for deletion
2020-11-26 11:14:56 -08:00
Chris Lu
eab53ea80d
filer leveldb store: a bit more efficient directory listing with prefix
2020-11-22 21:10:41 -08:00
Chris Lu
290b5e2cd0
directly delete file chunks
...
keeping current async deletions for now
2020-11-17 17:20:21 -08:00
Chris Lu
dc304342b2
fs.configure: configurable volume growth
2020-11-17 01:00:02 -08:00
Chris Lu
5f19e81dab
filer confi: support hierachical configuration
2020-11-16 16:50:12 -08:00
Chris Lu
c0d279c54e
filere.conf: prefer to use json format
2020-11-15 21:48:17 -08:00
Chris Lu
ee2fa14dbe
filer conf: delete location specific configuration
2020-11-15 20:15:47 -08:00
Chris Lu
71056dae07
fs.configure: read and local add filer configuration
2020-11-15 18:09:35 -08:00
Chris Lu
0ea5c087ce
go fmt
2020-11-15 16:59:28 -08:00
Chris Lu
95c0de285d
refactoring
2020-11-15 16:58:48 -08:00
Chris Lu
500bcab953
refactoring
2020-11-15 14:41:56 -08:00
Chris Lu
590f02179d
filer: load filer conf when starting
2020-11-15 14:06:03 -08:00
Chris Lu
0a406f652e
load filer conf and match by prefix
2020-11-15 00:26:05 -08:00
Chris Lu
0fc9ffc603
able to read filer.conf if updated
2020-11-14 21:21:20 -08:00
Chris Lu
442e092995
filer watch "/etc" folder for configuration changes
2020-11-14 14:26:08 -08:00
Konstantin Lebedev
aa30604cec
fix tests
2020-11-12 08:49:38 +05:00
Konstantin Lebedev
1eec5c8d5d
gen pb
2020-11-12 04:10:06 +05:00
Konstantin Lebedev
fc7baef5bb
fiil serverUrls sorted by data center
2020-11-12 02:13:33 +05:00
Chris Lu
8750cac090
move to util.RetryWaitTime
2020-11-01 02:36:43 -08:00
Chris Lu
df8d976bb0
refactoring
2020-11-01 01:58:48 -07:00
Chris Lu
16fae84414
fix format error
2020-10-24 20:11:31 -07:00
Chris Lu
e9d40b80b2
less verbose logs
2020-10-24 09:42:54 -07:00
Chris Lu
b81359823f
postgres: support empty user
2020-10-22 14:27:47 -07:00
Chris Lu
d75bc62196
to read files that are quickly changing
2020-10-22 12:59:15 -07:00
Chris Lu
5179e559f7
skip empty logs
2020-10-22 00:35:48 -07:00
Chris Lu
bd103c143a
add lock for vidCache
2020-10-21 19:28:59 -07:00
Chris Lu
68d39c86f1
mysql, postgres, cassandra: change kv key to base64 encoding
...
The exisitng key-value operation for stores using mysql, postgres, and maybe cassandra are already broken.
The kv is used to store hardlink, filer store signature and replication progress.
So users using hardlink and also uses mysql, postgres, or cassandra will have broken hard links.
Users using filer.sync will need to re-sync the files.
2020-10-16 11:10:12 -07:00
Chris Lu
c95d3d9dff
adjust tests
2020-10-14 12:27:52 -07:00
Chris Lu
93bcf56514
file read report EOF
...
fix https://github.com/chrislusf/seaweedfs/issues/1344
2020-10-14 12:18:24 -07:00
Chris Lu
0542911e29
go fmt
2020-10-13 19:50:46 -07:00
Chris Lu
58fa506491
minor
2020-10-13 19:50:22 -07:00
Chris Lu
aac4cb1f0c
adds errror on read and write
2020-10-13 13:53:34 -07:00
Chris Lu
3f7d1d1bf1
Only wait on retryable requests
2020-10-13 00:29:46 -07:00
Chris Lu
723ae11db4
refactoring in order to adjust volume server url later
2020-10-11 20:15:10 -07:00
Chris Lu
d155f907c2
mount: configurable read wait time
2020-10-10 20:09:43 -07:00
Chris Lu
8a52379ecb
add retry if volume can not be found
2020-10-10 16:02:39 -07:00
Chris Lu
70af0ec24c
filer: fix hanging on read
2020-10-10 16:02:10 -07:00
Chris Lu
cff8bb6554
return proper error
2020-10-10 15:43:22 -07:00
Chris Lu
6da8eef54f
fix logic error
2020-10-09 00:01:47 -07:00
Chris Lu
5d01dd28a6
add logging
2020-10-08 23:38:00 -07:00
Chris Lu
8d34eb0050
mount:exponentially backoff if read error for about 10 minutes
2020-10-08 23:31:26 -07:00
Chris Lu
6e1f936efd
refactoring
2020-10-08 23:19:42 -07:00
Chris Lu
b2ee5873fb
fix error not being returned
2020-10-08 23:19:20 -07:00
Chris Lu
8676db0f6a
adjust logging
2020-10-08 18:33:37 -07:00
Chris Lu
eed492b73b
randomize file locations
2020-10-07 23:58:32 -07:00
Chris Lu
62d3d3aea0
go fmt
2020-10-07 23:30:54 -07:00
Chris Lu
a8624c2e4f
read from alternative replica
...
related to https://github.com/chrislusf/seaweedfs/issues/1512
2020-10-07 22:49:04 -07:00
Chris Lu
36492c47ec
adjust
2020-10-05 14:06:18 -07:00
Chris Lu
8f8e9ddbde
adjust API
2020-10-05 09:47:07 -07:00
Konstantin Lebedev
e4f2d9eb4a
We return etag using the same algorithm as aws s3
...
https://teppen.io/2018/06/23/aws_s3_etags/
2020-10-05 14:43:32 +05:00
Chris Lu
6b591b02af
adjust log level
2020-10-04 16:21:43 -07:00
Chris Lu
666859f7cd
mount: read prefetching
2020-10-04 01:31:04 -07:00
Chris Lu
bc47835997
mount: avoid duplicated reads
2020-10-03 20:16:42 -07:00
Chris Lu
8d65ad1444
Revert "mount: adds read prefetching"
...
This reverts commit 692f0614d1
.
2020-10-03 19:40:56 -07:00
Chris Lu
692f0614d1
mount: adds read prefetching
2020-10-03 16:50:09 -07:00
limd
ed7816681a
Add cassandra authenticator mode
2020-09-25 11:22:47 +08:00
Chris Lu
0790c6d605
fix empty hard link id
2020-09-24 11:16:43 -07:00
Chris Lu
1012df7bb5
switch hardlink id from int64 to bytes
2020-09-24 11:11:42 -07:00
Chris Lu
2e7c361a0d
hardlink deletion factors in hardlink counter
2020-09-24 09:43:52 -07:00
Chris Lu
1295347958
adjust hardlink update
...
simplify logic, pass entity content directly to hard link. The "weed mount" handles the logic to calculate hard link counter.
2020-09-24 09:43:00 -07:00
Chris Lu
5e239afdfc
hardlink works now
2020-09-24 03:06:48 -07:00
Chris Lu
b61d33f251
expose only store wrapper to meta changes
2020-09-22 16:27:36 -07:00
Chris Lu
852e5f7cbc
filer: fix mongodb insert
...
fix https://github.com/chrislusf/seaweedfs/issues/1471
2020-09-17 21:50:52 -07:00
Chris Lu
10f9081526
filer: adjust meta data events to received timestamp
...
if a client is already connected and start from t0. A message recieved at t+1 but with timestamp t-1 may not be processed by the client.
This commit changes to the event received time, so the replication can be ordered.
2020-09-15 01:18:33 -07:00
Chris Lu
b0c7de186d
filer: fix postgres prefixed directory listing problem
...
fix https://github.com/chrislusf/seaweedfs/issues/1465
2020-09-12 13:37:03 -07:00
Chris Lu
446e476a11
go fmt
2020-09-12 04:08:03 -07:00
Chris Lu
2a0925590c
filer: etcd store avoid read with nil option
...
fix https://github.com/chrislusf/seaweedfs/issues/1463
2020-09-11 15:29:45 -07:00
ruitao.liu
5b0676049a
change elastic initialize process similar as others.
2020-09-10 23:35:20 +08:00
ruitao.liu
9be4e97625
change logs print format.
2020-09-10 16:30:15 +08:00
ruitao.liu
719dc43af1
modify elastic urls from string to array.
2020-09-10 16:24:09 +08:00
ruitao.liu
72f9d7f047
use util to generate md5.
2020-09-10 16:11:18 +08:00
ruitao.liu
6a5b38c0d4
fix elastic kv ops.
2020-09-10 15:59:16 +08:00
ruitao.liu
3f7fbfddca
add more basic elastic options.
2020-09-10 14:22:07 +08:00
Chris Lu
daf0a449f7
properly cancel context for streaming grpc
2020-09-09 12:07:15 -07:00
Chris Lu
387ab6796f
filer: cross cluster synchronization
2020-09-09 11:21:23 -07:00
Chris Lu
44b3f2efc8
filer: Elastic Search return correct kv error
...
fix https://github.com/chrislusf/seaweedfs/issues/1452
2020-09-07 12:44:51 -07:00
Chris Lu
d8af6f62df
adjust logging
2020-09-06 12:10:40 -07:00
Chris Lu
07f32feed0
print sync progress
2020-09-06 00:50:51 -07:00
Chris Lu
0af6252b2d
fix compilation
2020-09-06 00:50:38 -07:00
Chris Lu
ac494ff5e8
Update meta_aggregator.go
2020-09-06 00:29:16 -07:00
Chris Lu
c9f8f25ba5
read peer filer from start
2020-09-06 00:12:41 -07:00
Chris Lu
30dc365cbd
add self filer
2020-09-06 00:11:46 -07:00
Chris Lu
8acd714600
apply meta changes only if store is different
2020-09-05 23:25:03 -07:00
Chris Lu
d741ed66db
check whether shares the same filer store
2020-09-05 23:02:52 -07:00
Chris Lu
bba90ff3c8
read filer signature
2020-09-05 22:52:15 -07:00
Chris Lu
71b0e256c7
filer: setOrLoadFilerStoreSignature
2020-09-05 14:08:59 -07:00
ruitao.liu
450cf07505
skip the index that for kv usage.
2020-09-04 21:49:03 +08:00
ruitao.liu
1384ff9a2f
1.split kv in one file.
...
2.disable query for kv in es index.
2020-09-04 17:34:26 +08:00
ruitao.liu
83080b5e03
ES backended filer support kv ops.
2020-09-04 15:40:13 +08:00
Chris Lu
46f65a84a6
filer: elastic7 adjust default value
2020-09-03 20:12:38 -07:00
Chris Lu
f815fce81b
Merge pull request #1446 from taozix/master
...
new filer store option to es7.
2020-09-03 20:10:23 -07:00
Chris Lu
44b4ebf0ff
filer: Redis cleanly delete directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1448
2020-09-03 19:17:39 -07:00
Chris Lu
b8f32bcab9
filer: compress stored metadata
2020-09-03 11:00:20 -07:00
ruitao.liu
798280e980
change filer2 to filer.
2020-09-03 17:05:26 +08:00
ruitao.liu
a93d27d1e8
new filer option to es v7.
2020-09-03 16:34:58 +08:00
Chris Lu
68e878adb5
fix formatting
2020-09-02 21:42:12 -07:00
Chris Lu
0d99a5da3c
c*: support kv operations
2020-09-02 18:39:24 -07:00
Chris Lu
645a4af3db
mongodb: support kv operations
2020-09-02 17:19:14 -07:00
Chris Lu
ecaa30c408
better error message
2020-09-02 17:17:44 -07:00
Chris Lu
7c770b727c
Update abstract_sql_store_kv.go
2020-09-02 10:13:48 -07:00
Chris Lu
05a6e2dc7f
sql kv upsert
2020-09-02 10:12:44 -07:00
Chris Lu
ed62f52452
fix sql insert with duplicated primary key
2020-09-02 10:09:49 -07:00
Chris Lu
9ea290aa12
fix error message
2020-09-01 23:21:41 -07:00
Chris Lu
06a1188267
mysql/postgres: support kv operations
2020-09-01 22:48:36 -07:00
Chris Lu
aa40295f06
leveldb: report not found entry
2020-09-01 22:48:23 -07:00
Chris Lu
87d7312bf6
mysql/postgres: properly report entry not found
2020-09-01 22:47:57 -07:00
Chris Lu
72b0a5f1d1
mysql or postgres: log find error
2020-09-01 22:25:17 -07:00
Chris Lu
2815bbe6c0
go fmt
2020-09-01 21:59:26 -07:00
Chris Lu
37234bf3f8
filer store adds kv support
...
can compile now, need to implement those unimplemented
2020-09-01 21:59:00 -07:00
Chris Lu
2b14ae5819
fix tests
2020-09-01 01:33:43 -07:00
Chris Lu
d91ec535b3
fix tests
2020-09-01 01:29:13 -07:00
Chris Lu
eb7929a971
rename filer2 to filer
2020-09-01 00:21:19 -07:00
Chris Lu
8647191bee
removing filer package!
2018-05-27 11:52:15 -07:00
Chris Lu
9f345da20f
mv filer proto to filer_pb
2018-05-09 23:18:02 -07:00
Chris Lu
43a69d20bf
change filer API to gRPC
2018-05-08 01:59:43 -07:00
Chris Lu
2cada598c6
fix test
2018-05-05 15:08:23 -07:00
Chris Lu
fffb14bc87
better support FUSE Lookup()
2018-05-05 02:01:50 -07:00
Chris Lu
050ab19264
add design
2018-05-04 23:38:42 -07:00
Chris Lu
94a35f25f3
simple refactoring
2018-05-03 00:16:02 -07:00
Chris Lu
d49e91cd78
cassandra use local quorum for better performance
2018-04-16 01:33:05 -07:00
Chris Lu
15b8499f8d
refactoring
2018-04-16 01:16:06 -07:00
Chris Lu
8c64d06b3a
update redis library
2018-04-16 00:27:35 -07:00
chanyoung
52fc476a6f
filer: fixup print error message when make new directory
2017-08-15 15:54:12 +09:00
sparklxb
c46e91d229
complement weed mount: add read and delete
2017-03-17 15:02:27 +08:00
Stepan Bujnak
a72c92e55f
Added locking on the filer move operation
2017-03-11 02:52:04 -08:00
vancepym
ee6067e98c
Fix: the maximum-length character exceed 255
...
The length of a CHAR column is fixed to the length that you declare when you create the table. The length can be any value from 0 to 255. When CHAR values are stored, they are right-padded with spaces to the specified length.
see https://dev.mysql.com/doc/refman/5.7/en/char.html
2017-01-23 20:33:00 +08:00
Mike Tolman
43e655afc8
Updating PostgreSQL file store integration to support directories.
2016-12-12 13:03:32 -07:00
Mike Tolman
e88795be13
Adding PostgreSQL Support
2016-12-07 17:24:40 -07:00
Mike Tolman
b18ec59c56
Revert "Adding PostgreSQL Filer Support"
...
This reverts commit 10853e4d2f
.
2016-12-07 17:16:10 -07:00
Mike Tolman
10853e4d2f
Adding PostgreSQL Filer Support
2016-12-07 17:13:03 -07:00
Mike Tolman
a1783a14cc
Revert "Adding PostgreSQL Filer Support"
...
This reverts commit 855058858a
.
2016-12-07 16:54:06 -07:00
Mike Tolman
855058858a
Adding PostgreSQL Filer Support
2016-12-07 16:51:36 -07:00
Chris Lu
64509dd6ca
fix text.
2016-11-28 20:07:27 -08:00
eshujiushiwo
e025fc00a4
let filer use all cassandra server
...
let filer use all cassandra server
2016-11-28 18:23:39 +08:00
eshujiushiwo
5ee7fdc0cc
fix hosts strings
...
fix hosts strings
2016-11-28 16:03:39 +08:00
eshujiushiwo
3450eff288
fix info log
...
fix info log
2016-11-28 15:32:16 +08:00
eshujiushiwo
3e29d0f75b
Update cassandra_store.go
2016-11-28 15:14:19 +08:00
eshujiushiwo
809aa028ec
fix the connection of cassandra
...
fix the connection of cassandra ,let filer can connect to multi cassandra nodes which are in the one cluster
2016-11-28 15:01:04 +08:00
eshujiushiwo
89ccb6be05
Update cassandra_store.go
2016-11-28 12:16:51 +08:00
eshujiushiwo
5385bf9555
update cassandra connections
...
update cassandra connections
2016-11-28 11:27:40 +08:00
霍晓栋
78474409a5
filer mysqlstore bug fix
2016-09-08 11:35:54 +08:00
霍晓栋
3aa021a812
refactoring mysql store code
2016-09-05 14:10:22 +08:00
霍晓栋
e7b237c8da
UT case fix
2016-08-31 11:55:02 +08:00
霍晓栋
b0035747e3
add filer support
2016-08-31 11:32:30 +08:00
DaLin
fa2a7b5f2f
fix bug: filer DELETE
2016-06-11 01:16:19 +08:00
Chris Lu
5ce6bbf076
directory structure change to work with glide
...
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
2016-06-02 18:09:14 -07:00