Commit Graph

120 Commits

Author SHA1 Message Date
zemul
028ebb1d0d
Feat:merge small chunk (#6049)
* fix:mount deadlock

* feat: merge small chunk

* adjust MergeChunkMinCount

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2024-09-23 08:51:03 -07:00
Guang Jiong Lou
6c986e9d70
improve worm support (#5983)
* improve worm support

Signed-off-by: lou <alex1988@outlook.com>

* worm mode in filer

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* move to fs configure

Signed-off-by: lou <alex1988@outlook.com>

* remove flag

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* support worm hardlink

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* typo

Signed-off-by: lou <alex1988@outlook.com>

* sync filer conf

Signed-off-by: lou <alex1988@outlook.com>

---------

Signed-off-by: lou <alex1988@outlook.com>
2024-09-16 21:02:21 -07:00
vadimartynov
86d92a42b4
Added tls for http clients (#5766)
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
Konstantin Lebedev
f77eee667d
add s3test for sql (#5718)
* add s3test for sql

* fix test test_bucket_listv2_delimiter_basic for s3

* fix action s3tests

* regen s3 api xsd

* rm minor s3 test test_bucket_listv2_fetchowner_defaultempty

* add docs

* without xmlns
2024-07-04 11:00:41 -07:00
Konstantin Lebedev
5ffacbb6ea
refactor all methods strings to const (#5726) 2024-07-01 01:00:39 -07:00
chrislu
64fd233a59 DeleteUncommittedChunks 2024-06-15 09:55:22 -07:00
chrislu
677cfb8ad1 refactor 2024-06-15 09:39:49 -07:00
Konstantin Lebedev
8fb978d6f6
[volume] requests that were canceled by the client are not an internal server problem(500 => 499) (#5304) 2024-02-14 14:18:56 -08:00
julor
2fa37e0287 Update filer_server_handlers_write_autochunk.go 2023-11-20 11:17:14 -08:00
julor
9e03b916d8 verify content-md5 in header
if upload content by s3 client and set content-md5 in header,then will verify recived and saved content,if not equal delete thos chunks
2023-11-20 11:17:14 -08:00
Konstantin Lebedev
1cac5d983d
fix: disallow file name too long when writing a file (#4881)
* fix: disallow file name too long when writing a file

* bool LongerName to MaxFilenameLength

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-12 14:29:55 -07:00
chrislu
0bffb3770f support TTL when creating folders
related to https://github.com/seaweedfs/seaweedfs/discussions/4549
2023-06-05 23:29:25 -07:00
zemul
e9fda774f4
[Filer] post add param:saveInside (#4434)
* fix:mount deadlock

* feat: filer http upload to metadata

* feat: /etc save inside

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-29 08:31:05 -07:00
Rohit Chormale
59706c89fb
#4270 set http status code to 409 if dir already exists (#4287) 2023-03-08 07:45:08 -08:00
Chris Lu
d4566d4aaa
more solid weed mount (#4089)
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
2023-01-02 23:20:45 -08:00
famosss
15284e2bee
fix: Lost Content-Encoding metadata (#4010) 2022-11-23 22:52:34 -08:00
chrislu
70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks()
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
LHHDZ
d21e2f523d
split ExtAcpKey to ExtAmzOwnerKey and ExtAmzAclKey to avoid unn… (#3824)
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call

Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-11 20:14:14 -07:00
LHHDZ
3de1e19780
s3: sync bucket info from filer (#3759) 2022-09-29 12:29:01 -07:00
Konstantin Lebedev
5307cadb32
[s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF (#3597) 2022-09-05 07:47:43 -07:00
chrislu
f8fa430257 filer: retryable data chunk upload 2022-08-20 19:15:44 -07:00
chrislu
689b4ecdcc remove unused collection and replication from upload result 2022-08-20 18:14:57 -07:00
Konstantin Lebedev
22181dd018
refactor FilerRequest metrics (#3402)
* refactor FilerRequest metrics

* avoid double count proxy

* defer to
2022-08-04 01:44:54 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
4fd5f96598 filer: remove replication, collection, disk_type info from entry metadata
these metadata can change and are not used
2022-06-06 00:39:35 -07:00
chrislu
27732ecfa4 move s3 related constants from package http to s3_constants 2022-05-30 22:57:41 -07:00
chrislu
79b8e6a8c3 add a place holder for later merge consecutive chunks 2022-04-06 00:28:48 -07:00
zzq09494
9f16df736e fix-filer: calculation error of the method skipCheckParentDirEntry 2022-03-19 09:15:42 +08:00
chrislu
4042fdf3bb rename to skipCheckParentDir
related to https://github.com/chrislusf/seaweedfs/pull/2761

It's better to default to false.
2022-03-16 23:55:31 -07:00
zzq09494
40b0033fa7 go fmt 2022-03-17 14:19:48 +08:00
zzq09494
81cce4b4c3 filer: support uploading file without needEnsureParentDir 2022-03-17 10:53:47 +08:00
zzq09494
a6a8892255 Revert "filer: support uploading file without needEnsureParentDir"
This reverts commit a93c4947ba.
2022-03-17 10:27:17 +08:00
zzq09494
a93c4947ba filer: support uploading file without needEnsureParentDir 2022-03-17 10:18:23 +08:00
chrislu
bb0b784544 minor 2022-03-07 00:04:59 -08:00
banjiaojuhao
71f3046841 filer: add back isAppend function 2022-03-07 15:41:07 +08:00
chrislu
f1713c96ae avoid possible runtime error: index out of range [0] with length 0 2022-03-05 21:14:31 -08:00
banjiaojuhao
6ab09e9071 filer_http: support uploading file with offset 2022-02-22 00:15:00 +08:00
chrislu
7bf14452f7 s3: handle Response-Content-Disposition 2021-12-12 23:05:40 -08:00
Chris Lu
88ff8fc27b ensure uploaded chunks are deleted on error 2021-11-29 00:28:26 -08:00
Chris Lu
bea26549f2 avoid empty extended attribute value 2021-11-28 23:50:52 -08:00
Konstantin Lebedev
c78220a7f2 fix object tagging https://github.com/chrislusf/seaweedfs/issues/2389 2021-10-20 16:01:06 +05:00
Chris Lu
de1ec050e8 fier/s3: save "Content-Disposition" to extended properties
related to https://github.com/chrislusf/seaweedfs/issues/2371
2021-10-12 00:04:28 -07:00
Chris Lu
e9760f261e minor 2021-09-13 00:31:06 -07:00
Chris Lu
6923af7280 refactoring 2021-09-06 16:20:49 -07:00
Chris Lu
c6d4c16079 S3: add metadata with multipart upload
fix https://github.com/chrislusf/seaweedfs/issues/2173
2021-07-01 19:12:11 -07:00
Chris Lu
fb8036385a s3: save metadata during put-object
fix https://github.com/chrislusf/seaweedfs/issues/2092
2021-05-29 14:14:30 -07:00
Konstantin Lebedev
69c768870b - object write cache control
- object write expires
2021-05-24 15:43:55 +05:00
Chris Lu
519b0e1e49 filer: upload to a directory without "/" suffix
fix https://github.com/chrislusf/seaweedfs/issues/1988
2021-04-12 11:56:56 -07:00
Chris Lu
8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu
b5880334fc refactor 2021-03-30 21:07:34 -07:00