1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211221231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631641651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042052062072082092102112122132142152162172182192202212222232242252262272282292302312322332342352362372382392402412422432442452462472482492502512522532542552562572582592602612622632642652662672682692702712722732742752762772782792802812822832842852862872882892902912922932942952962972982993003013023033043053063073083093103113123133143153163173183193203213223233243253263273283293303313323333343353363373383393403413423433443453463473483493503513523533543553563573583593603613623633643653663673683693703713723733743753763773783793803813823833843853863873883893903913923933943953963973983994004014024034044054064074084094104114124134144154164174184194204214224234244254264274284294304314324334344354364374384394404414424434444454464474484494504514524534544554564574584594604614624634644654664674684694704714724734744754764774784794804814824834844854864874884894904914924934944954964974984995005015025035045055065075085095105115125135145155165175185195205215225235245255265275285295305315325335345355365375385395405415425435445455465475485495505515525535545555565575585595605615625635645655665675685695705715725735745755765775785795805815825835845855865875885895905915925935945955965975985996006016026036046056066076086096106116126136146156166176186196206216226236246256266276286296306316326336346356366376386396406416426436446456466476486496506516526536546556566576586596606616626636646656666676686696706716726736746756766776786796806816826836846856866876886896906916926936946956966976986997007017027037047057067077087097107117127137147157167177187197207217227237247257267277287297307317327337347357367377387397407417427437447457467477487497507517527537547557567577587597607617627637647657667677687697707717727737747757767777787797807817827837847857867877887897907917927937947957967977987998008018028038048058068078088098108118128138148158168178188198208218228238248258268278288298308318328338348358368378388398408418428438448458468478488498508518528538548558568578588598608618628638648658668678688698708718728738748758768778788798808818828838848858868878888898908918928938948958968978988999009019029039049059069079089099109119129139149159169179189199209219229239249259269279289299309319329339349359369379389399409419429439449459469479489499509519529539549559569579589599609619629639649659669679689699709719729739749759769779789799809819829839849859869879889899909919929939949959969979989991000100110021003100410051006100710081009101010111012101310141015101610171018101910201021102210231024102510261027102810291030103110321033103410351036103710381039104010411042104310441045104610471048104910501051105210531054105510561057105810591060106110621063106410651066106710681069107010711072107310741075107610771078107910801081108210831084108510861087108810891090109110921093109410951096109710981099110011011102110311041105110611071108110911101111111211131114111511161117111811191120112111221123112411251126112711281129113011311132113311341135113611371138113911401141114211431144114511461147114811491150115111521153115411551156115711581159116011611162116311641165116611671168116911701171117211731174117511761177117811791180118111821183118411851186118711881189119011911192119311941195119611971198119912001201120212031204120512061207120812091210121112121213121412151216121712181219122012211222122312241225122612271228122912301231123212331234123512361237123812391240124112421243124412451246124712481249125012511252125312541255125612571258125912601261126212631264126512661267126812691270127112721273127412751276127712781279128012811282128312841285128612871288128912901291129212931294129512961297129812991300130113021303130413051306130713081309131013111312131313141315131613171318131913201321132213231324132513261327132813291330133113321333133413351336133713381339134013411342134313441345134613471348134913501351135213531354135513561357135813591360136113621363136413651366136713681369137013711372137313741375137613771378137913801381 |
- _ _ ____ _
- ___| | | | _ \| |
- / __| | | | |_) | |
- | (__| |_| | _ <| |___
- \___|\___/|_| \_\_____|
- Things that could be nice to do in the future
- Things to do in project curl. Please tell us what you think, contribute and
- send us patches that improve things.
- Be aware that these are things that we could do, or have once been considered
- things we could do. If you want to work on any of these areas, please
- consider bringing it up for discussions first on the mailing list so that we
- all agree it is still a good idea for the project.
- All bugs documented in the KNOWN_BUGS document are subject for fixing.
- 1. libcurl
- 1.1 TFO support on Windows
- 1.2 Consult %APPDATA% also for .netrc
- 1.3 struct lifreq
- 1.4 alt-svc sharing
- 1.5 get rid of PATH_MAX
- 1.6 thread-safe sharing
- 1.8 CURLOPT_RESOLVE for any port number
- 1.9 Cache negative name resolves
- 1.10 auto-detect proxy
- 1.11 minimize dependencies with dynamically loaded modules
- 1.12 updated DNS server while running
- 1.13 c-ares and CURLOPT_OPENSOCKETFUNCTION
- 1.14 connect to multiple IPs in parallel
- 1.15 Monitor connections in the connection pool
- 1.16 Try to URL encode given URL
- 1.17 Add support for IRIs
- 1.18 try next proxy if one does not work
- 1.19 provide timing info for each redirect
- 1.20 SRV and URI DNS records
- 1.21 netrc caching and sharing
- 1.22 CURLINFO_PAUSE_STATE
- 1.23 Offer API to flush the connection pool
- 1.25 Expose tried IP addresses that failed
- 1.28 FD_CLOEXEC
- 1.29 WebSocket read callback
- 1.30 config file parsing
- 1.31 erase secrets from heap/stack after use
- 1.32 add asynch getaddrinfo support
- 1.33 make DoH inherit more transfer properties
- 2. libcurl - multi interface
- 2.1 More non-blocking
- 2.2 Better support for same name resolves
- 2.3 Non-blocking curl_multi_remove_handle()
- 2.4 Split connect and authentication process
- 2.5 Edge-triggered sockets should work
- 2.6 multi upkeep
- 2.7 Virtual external sockets
- 2.8 dynamically decide to use socketpair
- 3. Documentation
- 3.1 Improve documentation about fork safety
- 4. FTP
- 4.1 HOST
- 4.4 Support CURLOPT_PREQUOTE for directories listings
- 4.6 GSSAPI via Windows SSPI
- 4.7 STAT for LIST without data connection
- 4.8 Passive transfer could try other IP addresses
- 5. HTTP
- 5.1 Provide the error body from a CONNECT response
- 5.2 Obey Retry-After in redirects
- 5.3 Rearrange request header order
- 5.4 Allow SAN names in HTTP/2 server push
- 5.5 auth= in URLs
- 5.6 alt-svc should fallback if alt-svc does not work
- 5.7 Require HTTP version X or higher
- 6. TELNET
- 6.1 ditch stdin
- 6.2 ditch telnet-specific select
- 6.3 feature negotiation debug data
- 6.4 exit immediately upon connection if stdin is /dev/null
- 7. SMTP
- 7.1 Passing NOTIFY option to CURLOPT_MAIL_RCPT
- 7.2 Enhanced capability support
- 7.3 Add CURLOPT_MAIL_CLIENT option
- 8. POP3
- 8.2 Enhanced capability support
- 9. IMAP
- 9.1 Enhanced capability support
- 9.2 upload unread
- 10. LDAP
- 10.1 SASL based authentication mechanisms
- 10.2 CURLOPT_SSL_CTX_FUNCTION for LDAPS
- 10.3 Paged searches on LDAP server
- 10.4 Certificate-Based Authentication
- 11. SMB
- 11.1 File listing support
- 11.2 Honor file timestamps
- 11.3 Use NTLMv2
- 11.4 Create remote directories
- 12. FILE
- 12.1 Directory listing on non-POSIX
- 13. TLS
- 13.1 TLS-PSK with OpenSSL
- 13.2 TLS channel binding
- 13.3 Defeat TLS fingerprinting
- 13.4 Consider OCSP stapling by default
- 13.5 Export session ids
- 13.6 Provide callback for cert verification
- 13.7 Less memory massaging with Schannel
- 13.8 Support DANE
- 13.9 TLS record padding
- 13.10 Support Authority Information Access certificate extension (AIA)
- 13.11 Some TLS options are not offered for HTTPS proxies
- 13.13 Make sure we forbid TLS 1.3 post-handshake authentication
- 13.14 Support the clienthello extension
- 13.15 Select signature algorithms
- 13.16 Share the CA cache
- 13.17 Add missing features to TLS backends
- 15. Schannel
- 15.1 Extend support for client certificate authentication
- 15.2 Extend support for the --ciphers option
- 15.4 Add option to allow abrupt server closure
- 16. SASL
- 16.1 Other authentication mechanisms
- 16.2 Add QOP support to GSSAPI authentication
- 17. SSH protocols
- 17.1 Multiplexing
- 17.2 Handle growing SFTP files
- 17.3 Read keys from ~/.ssh/id_ecdsa, id_ed25519
- 17.4 Support CURLOPT_PREQUOTE
- 17.5 SSH over HTTPS proxy with more backends
- 17.6 SFTP with SCP:
- 18. Command line tool
- 18.1 sync
- 18.2 glob posts
- 18.4 --proxycommand
- 18.5 UTF-8 filenames in Content-Disposition
- 18.6 Option to make -Z merge lined based outputs on stdout
- 18.7 specify which response codes that make -f/--fail return error
- 18.9 Choose the name of file in braces for complex URLs
- 18.10 improve how curl works in a Windows console window
- 18.11 Windows: set attribute 'archive' for completed downloads
- 18.12 keep running, read instructions from pipe/socket
- 18.13 Acknowledge Ratelimit headers
- 18.14 --dry-run
- 18.15 --retry should resume
- 18.16 send only part of --data
- 18.17 consider filename from the redirected URL with -O ?
- 18.18 retry on network is unreachable
- 18.19 expand ~/ in config files
- 18.20 hostname sections in config files
- 18.21 retry on the redirected-to URL
- 18.23 Set the modification date on an uploaded file
- 18.24 Use multiple parallel transfers for a single download
- 18.25 Prevent terminal injection when writing to terminal
- 18.26 Custom progress meter update interval
- 18.27 -J and -O with %-encoded filenames
- 18.28 -J with -C -
- 18.29 --retry and transfer timeouts
- 19. Build
- 19.2 Enable PIE and RELRO by default
- 19.3 Do not use GNU libtool on OpenBSD
- 19.4 Package curl for Windows in a signed installer
- 19.5 make configure use --cache-file more and better
- 20. Test suite
- 20.1 SSL tunnel
- 20.2 nicer lacking perl message
- 20.3 more protocols supported
- 20.4 more platforms supported
- 20.6 Use the RFC 6265 test suite
- 20.8 Run web-platform-tests URL tests
- 21. MQTT
- 21.1 Support rate-limiting
- 21.2 Support MQTTS
- 21.3 Handle network blocks
- 22. TFTP
- 22.1 TFTP does not convert LF to CRLF for mode=netascii
- 23. Gopher
- 23.1 Handle network blocks
- ==============================================================================
- 1. libcurl
- 1.1 TFO support on Windows
- libcurl supports the CURLOPT_TCP_FASTOPEN option since 7.49.0 for Linux and
- macOS. Windows supports TCP Fast Open starting with Windows 10, version 1607
- and we should add support for it.
- TCP Fast Open is supported on several platforms but not on Windows. Work on
- this was once started but never finished.
- See https:
- 1.2 Consult %APPDATA% also for .netrc
- %APPDATA%\.netrc is not considered when running on Windows. should not it?
- See https:
- 1.3 struct lifreq
- Use 'struct lifreq' and SIOCGLIFADDR instead of 'struct ifreq' and
- SIOCGIFADDR on newer Solaris versions as they claim the latter is obsolete.
- To support IPv6 interface addresses for network interfaces properly.
- 1.4 alt-svc sharing
- The share interface could benefit from allowing the alt-svc cache to be
- possible to share between easy handles.
- See https:
- The share interface offers CURL_LOCK_DATA_CONNECT to have multiple easy
- handle share a connection cache, but due to how connections are used they are
- still not thread-safe when used shared.
- See https:
- The share interface offers CURL_LOCK_DATA_HSTS to have multiple easy handle
- share a HSTS cache, but this is not thread-safe.
- 1.5 get rid of PATH_MAX
- Having code use and rely on PATH_MAX is not nice:
- https:
- Currently the libssh2 SSH based code uses it, but to remove PATH_MAX from
- there we need libssh2 to properly tell us when we pass in a too small buffer
- and its current API (as of libssh2 1.2.7) does not.
- 1.6 thread-safe sharing
- Using the share interface users can share some data between easy handles but
- several of the sharing options are documented as not safe and supported to
- share between multiple concurrent threads. Fixing this would enable more
- users to share data in more powerful ways.
- 1.8 CURLOPT_RESOLVE for any port number
- This option allows applications to set a replacement IP address for a given
- host + port pair. Consider making support for providing a replacement address
- for the hostname on all port numbers.
- See https:
- 1.9 Cache negative name resolves
- A name resolve that has failed is likely to fail when made again within a
- short period of time. Currently we only cache positive responses.
- 1.10 auto-detect proxy
- libcurl could be made to detect the system proxy setup automatically and use
- that. On Windows, macOS and Linux desktops for example.
- The pull-request to use libproxy for this was deferred due to doubts on the
- reliability of the dependency and how to use it:
- https:
- libdetectproxy is a (C++) library for detecting the proxy on Windows
- https:
- 1.11 minimize dependencies with dynamically loaded modules
- We can create a system with loadable modules/plug-ins, where these modules
- would be the ones that link to 3rd party libs. That would allow us to avoid
- having to load ALL dependencies since only the necessary ones for this
- app/invoke/used protocols would be necessary to load. See
- https:
- 1.12 updated DNS server while running
- If /etc/resolv.conf gets updated while a program using libcurl is running, it
- is may cause name resolves to fail unless res_init() is called. We should
- consider calling res_init() + retry once unconditionally on all name resolve
- failures to mitigate against this. Firefox works like that. Note that Windows
- does not have res_init() or an alternative.
- https:
- 1.13 c-ares and CURLOPT_OPENSOCKETFUNCTION
- curl creates most sockets via the CURLOPT_OPENSOCKETFUNCTION callback and
- close them with the CURLOPT_CLOSESOCKETFUNCTION callback. However, c-ares
- does not use those functions and instead opens and closes the sockets itself.
- This means that when curl passes the c-ares socket to the
- CURLMOPT_SOCKETFUNCTION it is not owned by the application like other
- sockets.
- See https:
- 1.14 connect to multiple IPs in parallel
- curl currently implements the happy eyeball algorithm for connecting to the
- IPv4 and IPv6 alternatives for a host in parallel, sticking with the
- connection that "wins". We could implement a similar algorithm per individual
- IP family as well when there are multiple available addresses: start with the
- first address, then start a second attempt N milliseconds after and then a
- third another N milliseconds later. That way there would be less waiting when
- the first IP has problems. It also improves the connection timeout value
- handling for multiple address situations.
- 1.15 Monitor connections in the connection pool
- libcurl's connection cache or pool holds a number of open connections for the
- purpose of possible subsequent connection reuse. It may contain a few up to a
- significant amount of connections. Currently, libcurl leaves all connections
- as they are and first when a connection is iterated over for matching or
- reuse purpose it is verified that it is still alive.
- Those connections may get closed by the server side for idleness or they may
- get an HTTP/2 ping from the peer to verify that they are still alive. By
- adding monitoring of the connections while in the pool, libcurl can detect
- dead connections (and close them) better and earlier, and it can handle
- HTTP/2 pings to keep such ones alive even when not actively doing transfers
- on them.
- 1.16 Try to URL encode given URL
- Given a URL that for example contains spaces, libcurl could have an option
- that would try somewhat harder than it does now and convert spaces to %20 and
- perhaps URL encoded byte values over 128 etc (basically do what the redirect
- following code already does).
- https:
- 1.17 Add support for IRIs
- IRIs (RFC 3987) allow localized, non-ASCII, names in the URL. To properly
- support this, curl/libcurl would need to translate/encode the given input
- from the input string encoding into percent encoded output "over the wire".
- To make that work smoothly for curl users even on Windows, curl would
- probably need to be able to convert from several input encodings.
- 1.18 try next proxy if one does not work
- Allow an application to specify a list of proxies to try, and failing to
- connect to the first go on and try the next instead until the list is
- exhausted. Browsers support this feature at least when they specify proxies
- using PACs.
- https:
- 1.19 provide timing info for each redirect
- curl and libcurl provide timing information via a set of different
- time-stamps (CURLINFO_*_TIME). When curl is following redirects, those
- returned time value are the accumulated sums. An improvement could be to
- offer separate timings for each redirect.
- https:
- 1.20 SRV and URI DNS records
- Offer support for resolving SRV and URI DNS records for libcurl to know which
- server to connect to for various protocols (including HTTP).
- 1.21 netrc caching and sharing
- The netrc file is read and parsed each time a connection is setup, which
- means that if a transfer needs multiple connections for authentication or
- redirects, the file might be reread (and parsed) multiple times. This makes
- it impossible to provide the file as a pipe.
- 1.22 CURLINFO_PAUSE_STATE
- Return information about the transfer's current pause state, in both
- directions. https:
- 1.23 Offer API to flush the connection pool
- Sometimes applications want to flush all the existing connections kept alive.
- An API could allow a forced flush or just a forced loop that would properly
- close all connections that have been closed by the server already.
- 1.25 Expose tried IP addresses that failed
- When libcurl fails to connect to a host, it could offer the application the
- addresses that were used in the attempt. Source + dest IP, source + dest port
- and protocol (UDP or TCP) for each failure. Possibly as a callback. Perhaps
- also provide "reason".
- https:
- 1.28 FD_CLOEXEC
- It sets the close-on-exec flag for the file descriptor, which causes the file
- descriptor to be automatically (and atomically) closed when any of the
- exec-family functions succeed. Should probably be set by default?
- https:
- 1.29 WebSocket read callback
- Call the read callback once the connection is established to allow sending
- the first message in the connection.
- https:
- 1.30 config file parsing
- Consider providing an API, possibly in a separate companion library, for
- parsing a config file like curl's -K/--config option to allow applications to
- get the same ability to read curl options from files.
- See https:
- 1.31 erase secrets from heap/stack after use
- Introducing a concept and system to erase secrets from memory after use, it
- could help mitigate and lessen the impact of (future) security problems etc.
- However: most secrets are passed to libcurl as clear text from the
- application and then clearing them within the library adds nothing...
- https:
- 1.32 add asynch getaddrinfo support
- Use getaddrinfo_a() to provide an asynch name resolver backend to libcurl
- that does not use threads and does not depend on c-ares. The getaddrinfo_a
- function is (probably?) glibc specific but that is a widely used libc among
- our users.
- https:
- 1.33 make DoH inherit more transfer properties
- Some options are not inherited because they are not relevant for the DoH SSL
- connections, or inheriting the option may result in unexpected behavior. For
- example the user's debug function callback is not inherited because it would
- be unexpected for internal handles (ie DoH handles) to be passed to that
- callback.
- If an option is not inherited then it is not possible to set it separately
- for DoH without a DoH-specific option. For example:
- CURLOPT_DOH_SSL_VERIFYHOST, CURLOPT_DOH_SSL_VERIFYPEER and
- CURLOPT_DOH_SSL_VERIFYSTATUS.
- See https:
- 2. libcurl - multi interface
- 2.1 More non-blocking
- Make sure we do not ever loop because of non-blocking sockets returning
- EWOULDBLOCK or similar. Blocking cases include:
- - Name resolves on non-Windows unless c-ares or the threaded resolver is used.
- - The threaded resolver may block on cleanup:
- https:
- - file:
- - TELNET transfers
- - GSSAPI authentication for FTP transfers
- - The "DONE" operation (post transfer protocol-specific actions) for the
- protocols SFTP, SMTP, FTP. Fixing multi_done() for this is a worthy task.
- - curl_multi_remove_handle for any of the above. See section 2.3.
- - Calling curl_ws_send() from a callback
- 2.2 Better support for same name resolves
- If a name resolve has been initiated for name NN and a second easy handle
- wants to resolve that name as well, make it wait for the first resolve to end
- up in the cache instead of doing a second separate resolve. This is
- especially needed when adding many simultaneous handles using the same host
- name when the DNS resolver can get flooded.
- 2.3 Non-blocking curl_multi_remove_handle()
- The multi interface has a few API calls that assume a blocking behavior, like
- add_handle() and remove_handle() which limits what we can do internally. The
- multi API need to be moved even more into a single function that "drives"
- everything in a non-blocking manner and signals when something is done. A
- remove or add would then only ask for the action to get started and then
- multi_perform() etc still be called until the add/remove is completed.
- 2.4 Split connect and authentication process
- The multi interface treats the authentication process as part of the connect
- phase. As such any failures during authentication does not trigger the
- relevant QUIT or LOGOFF for protocols such as IMAP, POP3 and SMTP.
- 2.5 Edge-triggered sockets should work
- The multi_socket API should work with edge-triggered socket events. One of
- the internal actions that need to be improved for this to work perfectly is
- the 'maxloops' handling in transfer.c:readwrite_data().
- 2.6 multi upkeep
- In libcurl 7.62.0 we introduced curl_easy_upkeep. It unfortunately only works
- on easy handles. We should introduces a version of that for the multi handle,
- and also consider doing "upkeep" automatically on connections in the
- connection pool when the multi handle is in used.
- See https:
- 2.7 Virtual external sockets
- libcurl performs operations on the given file descriptor that presumes it is
- a socket and an application cannot replace them at the moment. Allowing an
- application to fully replace those would allow a larger degree of freedom and
- flexibility.
- See https:
- 2.8 dynamically decide to use socketpair
- For users who do not use curl_multi_wait() or do not care for
- curl_multi_wakeup(), we could introduce a way to make libcurl NOT
- create a socketpair in the multi handle.
- See https:
- 3. Documentation
- 3.1 Improve documentation about fork safety
- See https:
- 4. FTP
- 4.1 HOST
- HOST is a command for a client to tell which hostname to use, to offer FTP
- servers named-based virtual hosting:
- https:
- 4.4 Support CURLOPT_PREQUOTE for directions listings
- The lack of support is mostly an oversight and requires the FTP state machine
- to get updated to get fixed.
- https:
- 4.6 GSSAPI via Windows SSPI
- In addition to currently supporting the SASL GSSAPI mechanism (Kerberos V5)
- via third-party GSS-API libraries, such as Heimdal or MIT Kerberos, also add
- support for GSSAPI authentication via Windows SSPI.
- 4.7 STAT for LIST without data connection
- Some FTP servers allow STAT for listing directories instead of using LIST,
- and the response is then sent over the control connection instead of as the
- otherwise usedw data connection: https:
- This is not detailed in any FTP specification.
- 4.8 Passive transfer could try other IP addresses
- When doing FTP operations through a proxy at localhost, the reported spotted
- that curl only tried to connect once to the proxy, while it had multiple
- addresses and a failed connect on one address should make it try the next.
- After switching to passive mode (EPSV), curl could try all IP addresses for
- "localhost". Currently it tries ::1, but it should also try 127.0.0.1.
- See https:
- 5. HTTP
- 5.1 Provide the error body from a CONNECT response
- When curl receives a body response from a CONNECT request to a proxy, it
- always just reads and ignores it. It would make some users happy if curl
- instead optionally would be able to make that responsible available. Via a
- new callback? Through some other means?
- See https:
- 5.2 Obey Retry-After in redirects
- The Retry-After is said to dicate "the minimum time that the user agent is
- asked to wait before issuing the redirected request" and libcurl does not
- obey this.
- See https:
- 5.3 Rearrange request header order
- Server implementers often make an effort to detect browser and to reject
- clients it can detect to not match. One of the last details we cannot yet
- control in libcurl's HTTP requests, which also can be exploited to detect
- that libcurl is in fact used even when it tries to impersonate a browser, is
- the order of the request headers. I propose that we introduce a new option in
- which you give headers a value, and then when the HTTP request is built it
- sorts the headers based on that number. We could then have internally created
- headers use a default value so only headers that need to be moved have to be
- specified.
- 5.4 Allow SAN names in HTTP/2 server push
- curl only allows HTTP/2 push promise if the provided :authority header value
- exactly matches the hostname given in the URL. It could be extended to allow
- any name that would match the Subject Alternative Names in the server's TLS
- certificate.
- See https:
- 5.5 auth= in URLs
- Add the ability to specify the preferred authentication mechanism to use by
- using ;auth=<mech> in the login part of the URL.
- For example:
- http:
- --user test:pass;auth=NTLM or --user test:pass --ntlm from the command line.
- Additionally this should be implemented for proxy base URLs as well.
- 5.6 alt-svc should fallback if alt-svc does not work
- The alt-svc: header provides a set of alternative services for curl to use
- instead of the original. If the first attempted one fails, it should try the
- next etc and if all alternatives fail go back to the original.
- See https:
- 5.7 Require HTTP version X or higher
- curl and libcurl provide options for trying higher HTTP versions (for example
- HTTP/2) but then still allows the server to pick version 1.1. We could
- consider adding a way to require a minimum version.
- See https:
- 6. TELNET
- 6.1 ditch stdin
- Reading input (to send to the remote server) on stdin is a crappy solution
- for library purposes. We need to invent a good way for the application to be
- able to provide the data to send.
- 6.2 ditch telnet-specific select
- Move the telnet support's network select() loop go away and merge the code
- into the main transfer loop. Until this is done, the multi interface does not
- work for telnet.
- 6.3 feature negotiation debug data
- Add telnet feature negotiation data to the debug callback as header data.
- 6.4 exit immediately upon connection if stdin is /dev/null
- If it did, curl could be used to probe if there is an server there listening
- on a specific port. That is, the following command would exit immediately
- after the connection is established with exit code 0:
- curl -s --connect-timeout 2 telnet:
- 7. SMTP
- 7.1 Passing NOTIFY option to CURLOPT_MAIL_RCPT
- Is there a way to pass the NOTIFY option to the CURLOPT_MAIL_RCPT option ? I
- set a string that already contains a bracket. For instance something like
- that: curl_slist_append( recipients, "<foo@bar> NOTIFY=SUCCESS,FAILURE" );
- https:
- 7.2 Enhanced capability support
- Add the ability, for an application that uses libcurl, to obtain the list of
- capabilities returned from the EHLO command.
- 7.3 Add CURLOPT_MAIL_CLIENT option
- Rather than use the URL to specify the mail client string to present in the
- HELO and EHLO commands, libcurl should support a new CURLOPT specifically for
- specifying this data as the URL is non-standard and to be honest a bit of a
- hack ;-)
- Please see the following thread for more information:
- https:
- 8. POP3
- 8.2 Enhanced capability support
- Add the ability, for an application that uses libcurl, to obtain the list of
- capabilities returned from the CAPA command.
- 9. IMAP
- 9.1 Enhanced capability support
- Add the ability, for an application that uses libcurl, to obtain the list of
- capabilities returned from the CAPABILITY command.
- 9.2 upload unread
- Uploads over IMAP currently always set the email as "read" (or "seen"). It
- would be good to offer a way for users to select for uploads to remain
- unread.
- 10. LDAP
- 10.1 SASL based authentication mechanisms
- Currently the LDAP module only supports ldap_simple_bind_s() in order to bind
- to an LDAP server. However, this function sends username and password details
- using the simple authentication mechanism (as clear text). However, it should
- be possible to use ldap_bind_s() instead specifying the security context
- information ourselves.
- 10.2 CURLOPT_SSL_CTX_FUNCTION for LDAPS
- CURLOPT_SSL_CTX_FUNCTION works perfectly for HTTPS and email protocols, but
- it has no effect for LDAPS connections.
- https:
- 10.3 Paged searches on LDAP server
- https:
- 10.4 Certificate-Based Authentication
- LDAPS not possible with macOS and Windows with Certificate-Based Authentication
- https:
- 11. SMB
- 11.1 File listing support
- Add support for listing the contents of a SMB share. The output should
- probably be the same as/similar to FTP.
- 11.2 Honor file timestamps
- The timestamp of the transferred file should reflect that of the original
- file.
- 11.3 Use NTLMv2
- Currently the SMB authentication uses NTLMv1.
- 11.4 Create remote directories
- Support for creating remote directories when uploading a file to a directory
- that does not exist on the server, just like --ftp-create-dirs.
- 12. FILE
- 12.1 Directory listing on non-POSIX
- Listing the contents of a directory accessed with FILE only works on
- platforms with opendir. Support could be added for more systems, like
- Windows.
- 13. TLS
- 13.1 TLS-PSK with OpenSSL
- Transport Layer Security pre-shared key ciphersuites (TLS-PSK) is a set of
- cryptographic protocols that provide secure communication based on pre-shared
- keys (PSKs). These pre-shared keys are symmetric keys shared in advance among
- the communicating parties.
- https:
- 13.2 TLS channel binding
- TLS 1.2 and 1.3 provide the ability to extract some secret data from the TLS
- connection and use it in the client request (usually in some sort of
- authentication) to ensure that the data sent is bound to the specific TLS
- connection and cannot be successfully intercepted by a proxy. This
- functionality can be used in a standard authentication mechanism such as
- GSS-API or SCRAM, or in custom approaches like custom HTTP Authentication
- headers.
- For TLS 1.2, the binding type is usually tls-unique, and for TLS 1.3 it is
- tls-exporter.
- https:
- https:
- https:
- 13.3 Defeat TLS fingerprinting
- By changing the order of TLS extensions provided in the TLS handshake, it is
- sometimes possible to circumvent TLS fingerprinting by servers. The TLS
- extension order is of course not the only way to fingerprint a client.
- 13.4 Consider OCSP stapling by default
- Treat a negative response a reason for aborting the connection. Since OCSP
- stapling is presumed to get used much less in the future when Let's Encrypt
- drops the OCSP support, the benefit of this might however be limited.
- https:
- 13.5 Export session ids
- Add an interface to libcurl that enables "session IDs" to get
- exported/imported. Cris Bailiff said: "OpenSSL has functions which can
- serialise the current SSL state to a buffer of your choice, and recover/reset
- the state from such a buffer at a later date - this is used by mod_ssl for
- apache to implement and SSL session ID cache".
- 13.6 Provide callback for cert verification
- OpenSSL supports a callback for customised verification of the peer
- certificate, but this does not seem to be exposed in the libcurl APIs. Could
- it be? There is so much that could be done if it were.
- 13.7 Less memory massaging with Schannel
- The Schannel backend does a lot of custom memory management we would rather
- avoid: the repeated alloc + free in sends and the custom memory + realloc
- system for encrypted and decrypted data. That should be avoided and reduced
- for 1) efficiency and 2) safety.
- 13.8 Support DANE
- DNS-Based Authentication of Named Entities (DANE) is a way to provide SSL
- keys and certs over DNS using DNSSEC as an alternative to the CA model.
- https:
- An initial patch was posted by Suresh Krishnaswamy on March 7th 2013
- (https:
- approach. See Daniel's comments:
- https:
- correct library to base this development on.
- Björn Stenberg wrote a separate initial take on DANE that was never
- completed.
- 13.9 TLS record padding
- TLS (1.3) offers optional record padding and OpenSSL provides an API for it.
- I could make sense for libcurl to offer this ability to applications to make
- traffic patterns harder to figure out by network traffic observers.
- See https:
- 13.10 Support Authority Information Access certificate extension (AIA)
- AIA can provide various things like CRLs but more importantly information
- about intermediate CA certificates that can allow validation path to be
- fulfilled when the HTTPS server does not itself provide them.
- Since AIA is about downloading certs on demand to complete a TLS handshake,
- it is probably a bit tricky to get done right.
- See https:
- 13.11 Some TLS options are not offered for HTTPS proxies
- Some TLS related options to the command line tool and libcurl are only
- provided for the server and not for HTTPS proxies. --proxy-tls-max,
- --proxy-tlsv1.3, --proxy-curves and a few more.
- For more Documentation on this see:
- https:
- https:
- 13.13 Make sure we forbid TLS 1.3 post-handshake authentication
- RFC 8740 explains how using HTTP/2 must forbid the use of TLS 1.3
- post-handshake authentication. We should make sure to live up to that.
- See https:
- 13.14 Support the clienthello extension
- Certain stupid networks and middle boxes have a problem with SSL handshake
- packets that are within a certain size range because how that sets some bits
- that previously (in older TLS version) were not set. The clienthello
- extension adds padding to avoid that size range.
- https:
- https:
- 13.15 Select signature algorithms
- Consider adding an option or a way for users to select TLS signature
- algorithm. The signature algorithms set by a client are used directly in the
- supported signature algorithm in the client hello message.
- https:
- 13.16 Share the CA cache
- For TLS backends that supports CA caching, it makes sense to allow the share
- object to be used to store the CA cache as well via the share API. Would
- allow multiple easy handles to reuse the CA cache and save themselves from a
- lot of extra processing overhead.
- 13.17 Add missing features to TLS backends
- The feature matrix at https:
- features are supported by which TLS backends, and thus also where there are
- feature gaps.
- 15. Schannel
- 15.1 Extend support for client certificate authentication
- The existing support for the -E/--cert and --key options could be
- extended by supplying a custom certificate and key in PEM format, see:
- - Getting a Certificate for Schannel
- https:
- 15.2 Extend support for the --ciphers option
- The existing support for the --ciphers option could be extended
- by mapping the OpenSSL/GnuTLS cipher suites to the Schannel APIs, see
- - Specifying Schannel Ciphers and Cipher Strengths
- https:
- 15.4 Add option to allow abrupt server closure
- libcurl with Schannel errors without a known termination point from the server
- (such as length of transfer, or SSL "close notify" alert) to prevent against
- a truncation attack. Really old servers may neglect to send any termination
- point. An option could be added to ignore such abrupt closures.
- https:
- 16. SASL
- 16.1 Other authentication mechanisms
- Add support for other authentication mechanisms such as OLP,
- GSS-SPNEGO and others.
- 16.2 Add QOP support to GSSAPI authentication
- Currently the GSSAPI authentication only supports the default QOP of auth
- (Authentication), whilst Kerberos V5 supports both auth-int (Authentication
- with integrity protection) and auth-conf (Authentication with integrity and
- privacy protection).
- 17. SSH protocols
- 17.1 Multiplexing
- SSH is a perfectly fine multiplexed protocols which would allow libcurl to do
- multiple parallel transfers from the same host using the same connection,
- much in the same spirit as HTTP/2 does. libcurl however does not take
- advantage of that ability but does instead always create a new connection for
- new transfers even if an existing connection already exists to the host.
- To fix this, libcurl would have to detect an existing connection and "attach"
- the new transfer to the existing one.
- 17.2 Handle growing SFTP files
- The SFTP code in libcurl checks the file size *before* a transfer starts and
- then proceeds to transfer exactly that amount of data. If the remote file
- grows while the transfer is in progress libcurl does not notice and does not
- adapt. The OpenSSH SFTP command line tool does and libcurl could also just
- attempt to download more to see if there is more to get...
- https:
- 17.3 Read keys from ~/.ssh/id_ecdsa, id_ed25519
- The libssh2 backend in curl is limited to only reading keys from id_rsa and
- id_dsa, which makes it fail connecting to servers that use more modern key
- types.
- https:
- 17.4 Support CURLOPT_PREQUOTE
- The two other QUOTE options are supported for SFTP, but this was left out for
- unknown reasons.
- 17.5 SSH over HTTPS proxy with more backends
- The SSH based protocols SFTP and SCP did not work over HTTPS proxy at
- all until PR https:
- functionality with the libssh2 backend. Presumably, this support
- can/could be added for the other backends as well.
- 17.6 SFTP with SCP:
- OpenSSH 9 switched their 'scp' tool to speak SFTP under the hood. Going
- forward it might be worth having curl or libcurl attempt SFTP if SCP fails to
- follow suite.
- 18. Command line tool
- 18.1 sync
- "curl --sync http://example.com/feed[1-100].rss" or
- "curl --sync http://example.net/{index,calendar,history}.html"
- Downloads a range or set of URLs using the remote name, but only if the
- remote file is newer than the local file. A Last-Modified HTTP date header
- should also be used to set the mod date on the downloaded file.
- 18.2 glob posts
- Globbing support for -d and -F, as in 'curl -d "name=foo[0-9]" URL'.
- This is easily scripted though.
- 18.4 --proxycommand
- Allow the user to make curl run a command and use its stdio to make requests
- and not do any network connection by itself. Example:
- curl --proxycommand 'ssh pi@raspberrypi.local -W 10.1.1.75 80' \
- http:
- See https:
- 18.5 UTF-8 filenames in Content-Disposition
- RFC 6266 documents how UTF-8 names can be passed to a client in the
- Content-Disposition header, and curl does not support this.
- https:
- 18.6 Option to make -Z merge lined based outputs on stdout
- When a user requests multiple lined based files using -Z and sends them to
- stdout, curl does not "merge" and send complete lines fine but may send
- partial lines from several sources.
- https:
- 18.7 specify which response codes that make -f/--fail return error
- Allows a user to better specify exactly which error code(s) that are fine
- and which are errors for their specific uses cases
- 18.9 Choose the name of file in braces for complex URLs
- When using braces to download a list of URLs and you use complicated names
- in the list of alternatives, it could be handy to allow curl to use other
- names when saving.
- Consider a way to offer that. Possibly like
- {partURL1:name1,partURL2:name2,partURL3:name3} where the name following the
- colon is the output name.
- See https:
- 18.10 improve how curl works in a Windows console window
- If you pull the scrollbar when transferring with curl in a Windows console
- window, the transfer is interrupted and can get disconnected. This can
- probably be improved. See https:
- 18.11 Windows: set attribute 'archive' for completed downloads
- The archive bit (FILE_ATTRIBUTE_ARCHIVE, 0x20) separates files that shall be
- backed up from those that are either not ready or have not changed.
- Downloads in progress are neither ready to be backed up, nor should they be
- opened by a different process. Only after a download has been completed it is
- sensible to include it in any integer snapshot or backup of the system.
- See https:
- 18.12 keep running, read instructions from pipe/socket
- Provide an option that makes curl not exit after the last URL (or even work
- without a given URL), and then make it read instructions passed on a pipe or
- over a socket to make further instructions so that a second subsequent curl
- invoke can talk to the still running instance and ask for transfers to get
- done, and thus maintain its connection pool, DNS cache and more.
- 18.13 Acknowledge Ratelimit headers
- Consider a command line option that can make curl do multiple serial requests
- while acknowledging server specified rate limits:
- https:
- See https:
- 18.14 --dry-run
- A command line option that makes curl show exactly what it would do and send
- if it would run for real.
- See https:
- 18.15 --retry should resume
- When --retry is used and curl actually retries transfer, it should use the
- already transferred data and do a resumed transfer for the rest (when
- possible) so that it does not have to transfer the same data again that was
- already transferred before the retry.
- See https:
- 18.16 send only part of --data
- When the user only wants to send a small piece of the data provided with
- --data or --data-binary, like when that data is a huge file, consider a way
- to specify that curl should only send a piece of that. One suggested syntax
- would be: "--data-binary @largefile.zip!1073741823-2147483647".
- See https:
- 18.17 consider filename from the redirected URL with -O ?
- When a user gives a URL and uses -O, and curl follows a redirect to a new
- URL, the filename is not extracted and used from the newly redirected-to URL
- even if the new URL may have a much more sensible filename.
- This is clearly documented and helps for security since there is no surprise
- to users which filename that might get overwritten, but maybe a new option
- could allow for this or maybe -J should imply such a treatment as well as -J
- already allows for the server to decide what filename to use so it already
- provides the "may overwrite any file" risk.
- This is extra tricky if the original URL has no filename part at all since
- then the current code path does error out with an error message, and we
- cannot *know* already at that point if curl is redirected to a URL that has a
- filename...
- See https:
- 18.18 retry on network is unreachable
- The --retry option retries transfers on "transient failures". We later added
- --retry-connrefused to also retry for "connection refused" errors.
- Suggestions have been brought to also allow retry on "network is unreachable"
- errors and while totally reasonable, maybe we should consider a way to make
- this more configurable than to add a new option for every new error people
- want to retry for?
- https:
- 18.19 expand ~/ in config files
- For example .curlrc could benefit from being able to do this.
- See https:
- 18.20 hostname sections in config files
- config files would be more powerful if they could set different
- configurations depending on used URLs, hostname or possibly origin. Then a
- default .curlrc could a specific user-agent only when doing requests against
- a certain site.
- 18.21 retry on the redirected-to URL
- When curl is told to --retry a failed transfer and follows redirects, it
- might get an HTTP 429 response from the redirected-to URL and not the
- original one, which then could make curl decide to rather retry the transfer
- on that URL only instead of the original operation to the original URL.
- Perhaps extra emphasized if the original transfer is a large POST that
- redirects to a separate GET, and that GET is what gets the 529
- See https:
- 18.23 Set the modification date on an uploaded file
- For SFTP and possibly FTP, curl could offer an option to set the
- modification time for the uploaded file.
- See https:
- 18.24 Use multiple parallel transfers for a single download
- To enhance transfer speed, downloading a single URL can be split up into
- multiple separate range downloads that get combined into a single final
- result.
- An ideal implementation would not use a specified number of parallel
- transfers, but curl could:
- - First start getting the full file as transfer A
- - If after N seconds have passed and the transfer is expected to continue for
- M seconds or more, add a new transfer (B) that asks for the second half of
- A's content (and stop A at the middle).
- - If splitting up the work improves the transfer rate, it could then be done
- again. Then again, etc up to a limit.
- This way, if transfer B fails (because Range: is not supported) it lets
- transfer A remain the single one. N and M could be set to some sensible
- defaults.
- See https:
- 18.25 Prevent terminal injection when writing to terminal
- curl could offer an option to make escape sequence either non-functional or
- avoid cursor moves or similar to reduce the risk of a user getting tricked by
- clever tricks.
- See https:
- 18.26 Custom progress meter update interval
- Users who are for example doing large downloads in CI or remote setups might
- want the occasional progress meter update to see that the transfer is
- progressing and has not stuck, but they may not appreciate the
- many-times-a-second frequency curl can end up doing it with now.
- 18.27 -J and -O with %-encoded filenames
- -J/--remote-header-name does not decode %-encoded filenames. RFC 6266 details
- how it should be done. The can of worm is basically that we have no charset
- handling in curl and ASCII >=128 is a challenge for us. Not to mention that
- decoding also means that we need to check for nastiness that is attempted,
- like "../" sequences and the like. Probably everything to the left of any
- embedded slashes should be cut off.
- https:
- -O also does not decode %-encoded names, and while it has even less
- information about the charset involved the process is similar to the -J case.
- Note that we do not decode -O without the user asking for it with some other
- means, since -O has always been documented to use the name exactly as
- specified in the URL.
- 18.28 -J with -C -
- When using -J (with -O), automatically resumed downloading together with "-C
- -" fails. Without -J the same command line works. This happens because the
- resume logic is worked out before the target filename (and thus its
- pre-transfer size) has been figured out. This can be improved.
- https:
- 18.29 --retry and transfer timeouts
- If using --retry and the transfer timeouts (possibly due to using -m or
- -y/-Y) the next attempt does not resume the transfer properly from what was
- downloaded in the previous attempt but truncates and restarts at the original
- position where it was at before the previous failed attempt. See
- https:
- https:
- 19. Build
- 19.2 Enable PIE and RELRO by default
- Especially when having programs that execute curl via the command line, PIE
- renders the exploitation of memory corruption vulnerabilities a lot more
- difficult. This can be attributed to the additional information leaks being
- required to conduct a successful attack. RELRO, on the other hand, masks
- different binary sections like the GOT as read-only and thus kills a handful
- of techniques that come in handy when attackers are able to arbitrarily
- overwrite memory. A few tests showed that enabling these features had close
- to no impact, neither on the performance nor on the general functionality of
- curl.
- 19.3 Do not use GNU libtool on OpenBSD
- When compiling curl on OpenBSD with "--enable-debug" it gives linking errors
- when you use GNU libtool. This can be fixed by using the libtool provided by
- OpenBSD itself. However for this the user always needs to invoke make with
- "LIBTOOL=/usr/bin/libtool". It would be nice if the script could have some
- magic to detect if this system is an OpenBSD host and then use the OpenBSD
- libtool instead.
- See https:
- 19.4 Package curl for Windows in a signed installer
- See https:
- 19.5 make configure use --cache-file more and better
- The configure script can be improved to cache more values so that repeated
- invokes run much faster.
- See https:
- 20. Test suite
- 20.1 SSL tunnel
- Make our own version of stunnel for simple port forwarding to enable HTTPS
- and FTP-SSL tests without the stunnel dependency, and it could allow us to
- provide test tools built with either OpenSSL or GnuTLS
- 20.2 nicer lacking perl message
- If perl was not found by the configure script, do not attempt to run the tests
- but explain something nice why it does not.
- 20.3 more protocols supported
- Extend the test suite to include more protocols. The telnet could just do FTP
- or http operations (for which we have test servers).
- 20.4 more platforms supported
- Make the test suite work on more platforms. OpenBSD and macOS. Remove
- fork()s and it should become even more portable.
- 20.6 Use the RFC 6265 test suite
- A test suite made for HTTP cookies (RFC 6265) by Adam Barth is available at
- https:
- It would be good if someone would write a script/setup that would run curl
- with that test suite and detect deviances. Ideally, that would even be
- incorporated into our regular test suite.
- 20.8 Run web-platform-tests URL tests
- Run web-platform-tests URL tests and compare results with browsers on wpt.fyi
- It would help us find issues to fix and help us document where our parser
- differs from the WHATWG URL spec parsers.
- See https:
- 21. MQTT
- 21.1 Support rate-limiting
- The rate-limiting logic is done in the PERFORMING state in multi.c but MQTT
- is not (yet) implemented to use that.
- 21.2 Support MQTTS
- 21.3 Handle network blocks
- Running test suite with
- `CURL_DBG_SOCK_WBLOCK=90 ./runtests.pl -a mqtt` makes several
- MQTT test cases fail where they should not.
- 22. TFTP
- 22.1 TFTP does not convert LF to CRLF for mode=netascii
- RFC 3617 defines that an TFTP transfer can be done using "netascii"
- mode. curl does not support extracting that mode from the URL nor does it treat
- such transfers specifically. It should probably do LF to CRLF translations
- for them.
- See https:
- 23. Gopher
- 23.1 Handle network blocks
- Running test suite with
- `CURL_DBG_SOCK_WBLOCK=90 ./runtests.pl -a 1200 to 1300` makes several
- Gopher test cases fail where they should not.
|